Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration:
3.795022561s target:
https://us1.storj.io/login
FAIL (2025-09-06 09:30:39 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout
Duration: 3.795039244s
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
6 out of 60 is failed (threshold=3) [..................................................XXXXXX....] duration:
30.001337487s
FAIL (2025-09-06 09:30:38 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:43.779278335 (187h19m40.459473905s) [upload=164866]: Starting upload Sep 6 09:27:43.779338612 (187h19m40.459534181s) [upload=164866]: Done starting upload Sep 6 09:27:43.779387464 (187h19m40.459583033s) [upload=164866]: Uploading segments... Sep 6 09:27:43.779420899 (187h19m40.459616468s) [upload=164866]: Got next segment. Inline: true Sep 6 09:27:43.779424937 (187h19m40.459620506s) [upload=164866]: Waiting for error group managing segments... Sep 6 09:27:43.779457409 (187h19m40.459652978s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:43.779462638 (187h19m40.459658207s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:43.779465854 (187h19m40.459661413s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:53.418897371 (187h19m50.09909294s) [upload=164866]: Done uploading segments... Sep 6 09:27:53.418915256 (187h19m50.099110825s) [upload=164866]: Upload finished. err: <nil>
Duration: 30.001400628s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
6m21.283404493s
FAIL (2025-09-06 09:30:38 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6m21.283429681s
core
uplink
us1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
3 out of 60 is failed (threshold=3) [........................................................XXX.] duration:
23.128699642s target:
https://us1.storj.io/login
FAIL (2025-09-06 09:30:34 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout
Duration: 23.12872541s
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration:
37.237856146s
FAIL (2025-09-06 09:30:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0)
======LOGS======
oading piece...
Sep 6 09:29:59.994981184 (187h21m56.675176743s) [noise=true , address=45.91.116.11:4119 , piece_id=IS44MFMSZI4A4AMU5JDBRLNMEUL4EZTRXDRJYH5PE573I7SLR4ZQ, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:29:59.995034757 (187h21m56.675230326s) [noise=true , address=77.90.31.10:4517 , piece_id=66ZZDV5FIPTT7ERL264UIWMT6PSRKWQNZIK6SLYTC3VWSIAPEBSQ, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:29:59.995213830 (187h21m56.675409399s) [noise=true , address=45.91.116.7:4011 , piece_id=7TI3NEMJWFQ2ON4GQ4HNJD5RD4X3AIDC26TQO3ZU72OHFCCR6BOQ, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:29:59.995292890 (187h21m56.675488459s) [noise=true , address=77.90.31.16:4735 , piece_id=FUL5U2IHK2KMCZF6TB5HB3OSZ5VGOAA2I7V44QFF33WYZNY4VULQ, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:29:59.995486642 (187h21m56.675682211s) [noise=true , address=45.91.116.10:4081 , piece_id=CDTI3W4EXET6ZVWLTZ2R6DZU7QXZ2CRMI5JKXWLQKGPPIBI7HVXA, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:29:59.995536587 (187h21m56.675732156s) [noise=true , address=45.91.116.10:4083 , piece_id=VHDNWDKDFRTUA6YEMUZ3XC7LOST5CI3GWOQ6FB5TKAHBXQCHVHJA, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:29:59.995730898 (187h21m56.675926467s) [noise=true , address=45.91.116.6:4973 , piece_id=ZBABQ3QX2IU6I2CCLAGU2DHUB4HI3OEGMJJ7MMLRP3Q6EIIXDRAQ, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:29:59.995776837 (187h21m56.675972406s) [noise=true , address=45.91.116.9:4263 , piece_id=EOLE44HBBB45YAOC6QXSYQTNCEOVAVRSA4J5AIOPKF4PKZMOCLZQ, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:29:59.995947122 (187h21m56.676142691s) [noise=true , address=45.91.116.13:4153 , piece_id=4H5PX533ILJXP6DDSKQUEVQHJCHY7FYJEDXTSJXEVDQMOPOXB23A, seg_pos={0 0}, upload=164895]: Uploading piece...
Sep 6 09:30:21.764746582 (187h22m18.444942151s) [upload=164895]: Next segment err: context deadline exceeded
Sep 6 09:30:21.765993518 (187h22m18.446189087s) [noise=true , address=77.90.31.12:4589 , piece_id=3PMNYPEEOUOLGWNHLFUEM35QBQHKDW4O6C5ODSKT2XGNVQP25YBA, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:39288->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39288->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39288->77.90.31.12:4589: use of closed network connection
Sep 6 09:30:21.766055968 (187h22m18.446251537s) [noise=true , address=45.91.116.7:4009 , piece_id=M5P6XUILN4M43WET4E7HMSGNVMGJBEMIJM3PQJGK4FQNTAHK3GKA, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766137584 (187h22m18.446333153s) [noise=true , address=45.91.116.10:4081 , piece_id=CDTI3W4EXET6ZVWLTZ2R6DZU7QXZ2CRMI5JKXWLQKGPPIBI7HVXA, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766280056 (187h22m18.446475625s) [noise=true , address=45.91.116.7:4011 , piece_id=7TI3NEMJWFQ2ON4GQ4HNJD5RD4X3AIDC26TQO3ZU72OHFCCR6BOQ, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766324712 (187h22m18.446520281s) [noise=true , address=45.91.116.11:4119 , piece_id=IS44MFMSZI4A4AMU5JDBRLNMEUL4EZTRXDRJYH5PE573I7SLR4ZQ, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:52016->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766376972 (187h22m18.446572541s) [noise=true , address=77.90.31.10:4517 , piece_id=66ZZDV5FIPTT7ERL264UIWMT6PSRKWQNZIK6SLYTC3VWSIAPEBSQ, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:54510->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766424082 (187h22m18.446619651s) [noise=true , address=45.91.116.10:4083 , piece_id=VHDNWDKDFRTUA6YEMUZ3XC7LOST5CI3GWOQ6FB5TKAHBXQCHVHJA, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:52404->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766467695 (187h22m18.446663264s) [noise=true , address=45.91.116.8:4048 , piece_id=Z5EKTG43O57PW3AHI7D2FQC7B2LSDHI7ERLDRR5UODJI5IP7HX2Q, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45838->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766522941 (187h22m18.44671851s) [noise=true , address=77.90.31.16:4735 , piece_id=FUL5U2IHK2KMCZF6TB5HB3OSZ5VGOAA2I7V44QFF33WYZNY4VULQ, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:43464->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766528291 (187h22m18.44672386s) [noise=true , address=45.91.116.6:4975 , piece_id=YVPA6FA2NBS3U2TNMI2YPEM3EBT4LTGBJG2KWN5QPBINO4SXRC2A, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 188.166.225.170:44514->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:44514->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44514->45.91.116.6:4975: use of closed network connection
Sep 6 09:30:21.766562657 (187h22m18.446758226s) [noise=true , address=45.91.116.9:4263 , piece_id=EOLE44HBBB45YAOC6QXSYQTNCEOVAVRSA4J5AIOPKF4PKZMOCLZQ, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:53076->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766576504 (187h22m18.446772073s) [noise=true , address=77.90.31.12:4592 , piece_id=ZOLK25TANBQRTVYKKHZUZVMU6YUEQ27IBTY6NEQLQYBN3WLV2GHQ, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:59806->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766584509 (187h22m18.446780088s) [noise=true , address=77.90.31.13:4627 , piece_id=SGLAF44DPW27HUNBPE54DV3NBCZZBD536LML3CV32CVAHLR4DLCQ, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:39248->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766592064 (187h22m18.446787633s) [noise=true , address=45.91.116.13:4153 , piece_id=4H5PX533ILJXP6DDSKQUEVQHJCHY7FYJEDXTSJXEVDQMOPOXB23A, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 188.166.225.170:45270->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766608915 (187h22m18.446804485s) [noise=true , address=45.91.116.6:4973 , piece_id=ZBABQ3QX2IU6I2CCLAGU2DHUB4HI3OEGMJJ7MMLRP3Q6EIIXDRAQ, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:36554->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766614135 (187h22m18.446809704s) [noise=true , address=45.91.116.5:4298 , piece_id=3EZCODYRPMMVS3L6A56522MR3CTZ4X4DQTYT4H4MKZET4XG3NAJA, seg_pos={0 0}, upload=164895]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:50616->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:21.766647891 (187h22m18.44684346s) [seg_pos={0 0}, upload=164895]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil>
Duration: 37.23787343s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
1m2.333692308s
FAIL (2025-09-06 09:30:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0)
======LOGS======
of closed network connection; write tcp 188.166.225.170:40874->145.79.227.14:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40874->145.79.227.14:4854: use of closed network connection
Sep 6 09:30:22.351952654 (187h22m19.032148223s) [noise=true , address=145.79.227.44:4209 , piece_id=6OOXI7EWLZ2RDTB6B52BD4ZQOY3JDODIJEUYE42GHHCTTENP6RQA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR, address:145.79.227.44:4209): protocol: write tcp 188.166.225.170:47208->145.79.227.44:4209: use of closed network connection; write tcp 188.166.225.170:47208->145.79.227.44:4209: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47208->145.79.227.44:4209: use of closed network connection
Sep 6 09:30:22.352023309 (187h22m19.032218878s) [noise=true , address=145.79.227.17:4992 , piece_id=XGBU57LVSMTOGWPNJPDLNFTPL5RFNKD2KI67ZVNKBQA6EASN7EAQ, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:145.79.227.17:4992): protocol: write tcp 188.166.225.170:40722->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:40722->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40722->145.79.227.17:4992: use of closed network connection
Sep 6 09:30:22.352078796 (187h22m19.032274365s) [noise=true , address=145.79.227.37:4579 , piece_id=WWPMUOCD7XKHHBZ2A3K34CKWDRYHFKCLMGDGH4IJBQLBTTD625TA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:12ZMdnhz2eNgqaF2DpsJmyb6xJQKFYvmitNy6fqMugcnsuZqmak, address:145.79.227.37:4579): protocol: write tcp 188.166.225.170:36240->145.79.227.37:4579: use of closed network connection; write tcp 188.166.225.170:36240->145.79.227.37:4579: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36240->145.79.227.37:4579: use of closed network connection
Sep 6 09:30:22.352169629 (187h22m19.032365198s) [noise=true , address=77.90.31.21:4882 , piece_id=XEM6C37SF5UU2RYJTAI3NEOFOFGEMPNEA6WWIG3M7Q3TVKRT5EGA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 188.166.225.170:36186->77.90.31.21:4882: use of closed network connection; write tcp 188.166.225.170:36186->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36186->77.90.31.21:4882: use of closed network connection
Sep 6 09:30:22.352194567 (187h22m19.032390136s) [noise=true , address=45.91.116.4:4946 , piece_id=YH4UNKNRYMYNFBBJ6T2G7ZJAFUHA6QYGPXKDUQBWCLKQK3Q7PAXQ, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve, address:45.91.116.4:4946): protocol: write tcp 188.166.225.170:58756->45.91.116.4:4946: use of closed network connection; write tcp 188.166.225.170:58756->45.91.116.4:4946: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58756->45.91.116.4:4946: use of closed network connection
Sep 6 09:30:22.352241658 (187h22m19.032437237s) [noise=true , address=145.79.227.4:4190 , piece_id=PNNX3GEIYP75GZNBW4LJIWFDMLGK2UBLKYJZLPJZXKANCYIJAUXA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:1eNJzs5Dc7Nv7imoQXZDp2mEVWjD8o4viGVkFvuAeZ1arynxu4, address:145.79.227.4:4190): protocol: write tcp 188.166.225.170:35580->145.79.227.4:4190: use of closed network connection; write tcp 188.166.225.170:35580->145.79.227.4:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35580->145.79.227.4:4190: use of closed network connection
Sep 6 09:30:22.352346658 (187h22m19.032542227s) [noise=true , address=77.90.31.13:4649 , piece_id=P7ZTPM427UWWWNERW5BWB5YZ2BONTXLGTABVSTXY7QFY2XYPYG6A, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: read tcp 188.166.225.170:43672->77.90.31.13:4649: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:22.352395321 (187h22m19.0325909s) [noise=true , address=145.79.227.15:4892 , piece_id=FFS4K3CG7WCYTY3ZOAQ4IX3O4QMWXMMSKVIEAEUVRSKJWOIN63PA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:12MmZivyjXMMHcDgU3aG4HkWqBgY2QRf4Q1pkxJDF5QKSTQ7RXA, address:145.79.227.15:4892): protocol: read tcp 188.166.225.170:39264->145.79.227.15:4892: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:22.352467800 (187h22m19.032663379s) [noise=true , address=145.79.227.3:4178 , piece_id=FFFOZXJ6ERWCRRUNVWWGXQ3HE4PYFTORCQSHXKBU4G6HUWJ4ZUFA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:12LjJ3SmaNTQi8U1rPSGmR461UYC2CfMtoZ22SSV97hDuhMwC4V, address:145.79.227.3:4178): protocol: read tcp 188.166.225.170:40712->145.79.227.3:4178: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:22.352482818 (187h22m19.032678397s) [noise=true , address=45.91.116.13:4165 , piece_id=HDSJ2CJXTQD4V7A6CJOPDRC3E7F44LM2OISBTQO2YWBY473ZIVUQ, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 188.166.225.170:33982->45.91.116.13:4165: use of closed network connection; write tcp 188.166.225.170:33982->45.91.116.13:4165: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33982->45.91.116.13:4165: use of closed network connection
Sep 6 09:30:22.352615833 (187h22m19.032811402s) [noise=true , address=145.79.227.52:4447 , piece_id=3ABY7775RGNICFQCJ7PQMREGNRD4BLN3QR7XX5UXNBUH75SWKPTA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: read tcp 188.166.225.170:41094->145.79.227.52:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:22.352658876 (187h22m19.032854445s) [noise=true , address=145.79.227.11:4761 , piece_id=P4T52L3P7HNZAY7WGCVWELGSCMOXO4XEKGLZNBLV2OAGNG5BINQQ, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:12CXHkaBwi15pBUoRiZHi6JP5rBDu1iVpWT1WFL3dCmk8LRRxqw, address:145.79.227.11:4761): protocol: write tcp 188.166.225.170:60786->145.79.227.11:4761: use of closed network connection; write tcp 188.166.225.170:60786->145.79.227.11:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60786->145.79.227.11:4761: use of closed network connection
Sep 6 09:30:22.352683713 (187h22m19.032879282s) [noise=true , address=145.79.227.49:4344 , piece_id=V3ZALJVUHNXFA22WU42BD6OUUMBXOL55SFSEXDUPBPQTXSDHZSZQ, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: read tcp 188.166.225.170:58348->145.79.227.49:4344: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:22.352709152 (187h22m19.032904721s) [noise=true , address=145.79.227.26:4657 , piece_id=ECNOXI47B3WLDPIRH7UGEIWPZGX3RVQFHBXWZQNEZCXA4K6I5OCQ, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:129MyUjh5Nw3NhQxefnBzxYUK3mSXqwpW3nLa3KerYrjpuJiUfq, address:145.79.227.26:4657): protocol: write tcp 188.166.225.170:42682->145.79.227.26:4657: use of closed network connection; write tcp 188.166.225.170:42682->145.79.227.26:4657: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42682->145.79.227.26:4657: use of closed network connection
Sep 6 09:30:22.352744550 (187h22m19.032940119s) [noise=true , address=145.79.227.36:4527 , piece_id=VBPUNETVTPVGVJLDNTR4U6JNBYGYDQ4I2UD4HNNSMTXR4K3M5LOQ, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:145.79.227.36:4527): protocol: read tcp 188.166.225.170:51248->145.79.227.36:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:22.352759158 (187h22m19.032954727s) [noise=true , address=77.90.31.18:4817 , piece_id=TEMPDIL6TV3NNCZVXCDEQJZC6ODUGQITUASJBKTQXZZN3GJLGSTA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 188.166.225.170:55762->77.90.31.18:4817: use of closed network connection; write tcp 188.166.225.170:55762->77.90.31.18:4817: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55762->77.90.31.18:4817: use of closed network connection
Sep 6 09:30:22.352806979 (187h22m19.033002548s) [noise=true , address=77.90.31.17:4794 , piece_id=L7E67RM23MMYYOX3UNDGLB6DS6ARMSIAGDSRCBVW47327DACU2SA, seg_pos={0 0}, upload=164886]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: read tcp 188.166.225.170:52944->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:30:22.352869028 (187h22m19.033064597s) [seg_pos={0 0}, upload=164886]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil>
Duration: 1m2.33371399s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
3.307376122s
FAIL (2025-09-06 09:30:17 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 3.307406761s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X..X....] duration:
5.316993422s
FAIL (2025-09-06 09:30:12 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 48)
Duration: 5.317025394s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................XX...] duration:
25.07624561s
FAIL (2025-09-06 09:30:10 UTC)
orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:05.239202080 (187h19m1.919397649s) [upload=164859]: Starting upload Sep 6 09:27:05.239268026 (187h19m1.919463595s) [upload=164859]: Done starting upload Sep 6 09:27:05.239318563 (187h19m1.919514132s) [upload=164859]: Uploading segments... Sep 6 09:27:05.239355453 (187h19m1.919551022s) [upload=164859]: Got next segment. Inline: true Sep 6 09:27:05.239359901 (187h19m1.91955547s) [upload=164859]: Waiting for error group managing segments... Sep 6 09:27:05.239392204 (187h19m1.919587773s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:05.239396332 (187h19m1.919591901s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:05.239398846 (187h19m1.919594415s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:09.107329409 (187h19m5.787524978s) [upload=164859]: Done uploading segments... Sep 6 09:27:09.107348155 (187h19m5.787543714s) [upload=164859]: Upload finished. err: <nil>
Duration: 25.076262592s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
6 out of 60 is failed (threshold=3) [...................................................XXXXXX...] duration:
25.583768194s
FAIL (2025-09-06 09:30:04 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:43.779278335 (187h19m40.459473905s) [upload=164866]: Starting upload Sep 6 09:27:43.779338612 (187h19m40.459534181s) [upload=164866]: Done starting upload Sep 6 09:27:43.779387464 (187h19m40.459583033s) [upload=164866]: Uploading segments... Sep 6 09:27:43.779420899 (187h19m40.459616468s) [upload=164866]: Got next segment. Inline: true Sep 6 09:27:43.779424937 (187h19m40.459620506s) [upload=164866]: Waiting for error group managing segments... Sep 6 09:27:43.779457409 (187h19m40.459652978s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:43.779462638 (187h19m40.459658207s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:43.779465854 (187h19m40.459661413s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:53.418897371 (187h19m50.09909294s) [upload=164866]: Done uploading segments... Sep 6 09:27:53.418915256 (187h19m50.099110825s) [upload=164866]: Upload finished. err: <nil>
Duration: 25.583795796s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration:
17.6344472s
FAIL (2025-09-06 09:30:03 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 17.634480595s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
7.730388995s
FAIL (2025-09-06 09:30:00 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 7.730415918s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration:
30.000872048s
FAIL (2025-09-06 09:29:51 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 6 09:29:28.379414238 (187h21m25.059609807s) [upload=164887]: Starting upload Sep 6 09:29:28.379470345 (187h21m25.059665914s) [upload=164887]: Done starting upload Sep 6 09:29:28.379548615 (187h21m25.059744184s) [upload=164887]: Uploading segments... Sep 6 09:29:28.379664627 (187h21m25.059860196s) [upload=164887]: Got next segment. Inline: false Sep 6 09:29:28.379700144 (187h21m25.059895713s) [upload=164887]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:29:28.379704112 (187h21m25.059899681s) [upload=164887]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 6 09:29:51.508748492 (187h21m48.188944061s) [upload=164887]: Next segment err: context deadline exceeded Sep 6 09:29:51.508917085 (187h21m48.189112654s) [upload=164887]: Done uploading segments... Sep 6 09:29:51.508942543 (187h21m48.189138132s) [upload=164887]: Upload finished. err: context deadline exceeded
Duration: 30.000916684s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
1m0.174934547s
FAIL (2025-09-06 09:29:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 8)
======LOGS======
nection
Sep 6 09:29:51.807163851 (187h21m48.48735942s) [noise=true , address=81.171.14.70:4038 , piece_id=JMC5BHWU7MHVJZ7CRYA7KM7OJ3D67FY35KOS7NV5EDME3AVWZFSQ, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: write tcp 188.166.225.170:51740->81.171.14.70:4038: use of closed network connection; write tcp 188.166.225.170:51740->81.171.14.70:4038: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51740->81.171.14.70:4038: use of closed network connection
Sep 6 09:29:51.807233996 (187h21m48.487429565s) [noise=true , address=81.171.14.70:4032 , piece_id=7TZYBAQXEJLJ2WM7DZO7M43DTBEJHMEF5H2QVGTLPINVQ4TL4C7Q, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf, address:81.171.14.70:4032): protocol: write tcp 188.166.225.170:47462->81.171.14.70:4032: use of closed network connection; write tcp 188.166.225.170:47462->81.171.14.70:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47462->81.171.14.70:4032: use of closed network connection
Sep 6 09:29:51.807366189 (187h21m48.487561758s) [noise=true , address=185.167.204.205:4010 , piece_id=F2NGRGSFQW5AAFTZM46ZIMLGR7ZCZF72JKISSNPYUEECRAZAADXA, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: write tcp 188.166.225.170:45330->185.167.204.205:4010: use of closed network connection; write tcp 188.166.225.170:45330->185.167.204.205:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45330->185.167.204.205:4010: use of closed network connection
Sep 6 09:29:51.807388331 (187h21m48.487583901s) [noise=true , address=81.171.14.70:4025 , piece_id=IHDXNGVTN2XNJXCG7642ZAAYR7GBWNUNE2ZX6AK3DFI4XHEN2P7A, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, address:81.171.14.70:4025): protocol: write tcp 188.166.225.170:57988->81.171.14.70:4025: use of closed network connection; write tcp 188.166.225.170:57988->81.171.14.70:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57988->81.171.14.70:4025: use of closed network connection
Sep 6 09:29:51.807710357 (187h21m48.487905926s) [noise=true , address=185.167.204.206:4063 , piece_id=E2M7H2Z2JRAARFRDCIYJUWLAG4FNRCJ4ZHXOG2FK42UGGK66INRQ, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: write tcp 188.166.225.170:41288->185.167.204.206:4063: use of closed network connection; write tcp 188.166.225.170:41288->185.167.204.206:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41288->185.167.204.206:4063: use of closed network connection
Sep 6 09:29:51.807755333 (187h21m48.487950902s) [noise=true , address=81.171.14.70:4048 , piece_id=E4IMKATEAYFEZB4OEL4HP3NWYVX2D5YP4FTWOYFNMTBREI5XONWQ, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:48984->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:48984->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48984->81.171.14.70:4048: use of closed network connection
Sep 6 09:29:51.807819656 (187h21m48.488015225s) [noise=true , address=185.167.204.206:4069 , piece_id=FUBZINBRWMLX2J2Q5K5KPJUXILK3SGFSKHD52ZNAHKTDNKWJBLIA, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: read tcp 188.166.225.170:39794->185.167.204.206:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.807842109 (187h21m48.488037678s) [noise=true , address=185.167.204.206:4066 , piece_id=7BMRTKBCOS4VBCCHQYL7UZ4FOYEAL7BCCTA3V6DNYR6YH2XSWEIQ, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: read tcp 188.166.225.170:47418->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.807853191 (187h21m48.48804876s) [noise=true , address=81.171.14.70:4044 , piece_id=ZDKRUKFTKQQTLF6LO5C5PEGKKNUDLJ2DSAKL4VJ4SZBCEDUNDVAA, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr, address:81.171.14.70:4044): protocol: read tcp 188.166.225.170:49174->81.171.14.70:4044: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.807882317 (187h21m48.488077886s) [noise=true , address=81.171.14.70:4026 , piece_id=LDINX3MEJKUCYYUXTJKNXKIBOZRVK7GURHJ264K45STK5L7GPIXQ, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: read tcp 188.166.225.170:47154->81.171.14.70:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.807912184 (187h21m48.488107753s) [noise=true , address=185.167.204.206:4049 , piece_id=YEAMMSODER7UHRBM7ZWZVTBZJSFJPMDO6YKVSJH7QE3RJKGYYBGA, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:12VQJie7zan6SRJRmRm36gsVfY8UsYqtJeFRgKZy6m5d6B1aLFM, address:185.167.204.206:4049): protocol: read tcp 188.166.225.170:36530->185.167.204.206:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.807921241 (187h21m48.48811681s) [noise=true , address=185.167.204.206:4054 , piece_id=U6JALCNJPNSSFNRLURP55HX7AIHBIPDQGZQT2ZOV3YMYHIDI5YDQ, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: read tcp 188.166.225.170:59528->185.167.204.206:4054: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.807940017 (187h21m48.488135586s) [noise=true , address=185.167.204.205:4015 , piece_id=TAGRIXRXL5EFDI5UR3FAJQIXI77U3Q7RVMYYC3TNQJWFOMCDC7GQ, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:1pAUTtyVrPw8FCi8oTE1dsU8avWdehnDdCdtYN4P3zedYoBfiX, address:185.167.204.205:4015): protocol: read tcp 188.166.225.170:53616->185.167.204.205:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.807969624 (187h21m48.488165193s) [noise=true , address=81.171.14.70:4040 , piece_id=A3IAFPXODCDJPBH6WMR2OK3J3CNQ7E27AGUTHFESEJSDVD76OKDA, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: read tcp 188.166.225.170:55252->81.171.14.70:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.807998098 (187h21m48.488193667s) [noise=true , address=185.167.204.206:4062 , piece_id=XLRHZP4TIPP6Z5J4Q2ZRRTXM2MKKZYB72SN6LX5VCJ2HSZ2UUV3Q, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: read tcp 188.166.225.170:44750->185.167.204.206:4062: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.808023537 (187h21m48.488219106s) [noise=true , address=185.167.204.205:4035 , piece_id=C4YILC2M4S4CRK6AF3HOBMEO3AGVJVCLDXATBAW66AM3HRXSQGTA, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: read tcp 188.166.225.170:38430->185.167.204.205:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.808031592 (187h21m48.488227161s) [noise=true , address=185.167.204.206:4040 , piece_id=JNQFG2JUVUZRZSF7DDCE7ZW2MHYGDLLT7J4ELFW36S27JASLK64Q, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:1F6P8VfgAdXdcXykRHWpik5CDN4aWj7LKrugb8uAThZpAchkdT, address:185.167.204.206:4040): protocol: read tcp 188.166.225.170:46150->185.167.204.206:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.808047503 (187h21m48.488243072s) [noise=true , address=81.171.14.70:4041 , piece_id=RHA5PBCCUPQPPG5UUNUYP437J2AYDGC7PXQUNXW3WYONGMW7K5QQ, seg_pos={0 0}, upload=164881]: Done uploading piece. err: ecclient: upload failed (node:12U7CWC6tPDsNkmYf6uyKgFjwvYaUwUte6dmzUsZQPzm7ecPsLC, address:81.171.14.70:4041): protocol: read tcp 188.166.225.170:57420->81.171.14.70:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:51.808098550 (187h21m48.48829412s) [seg_pos={0 0}, upload=164881]: Done waiting for segment. successful: 8 optimal: 26 errs: <nil>
Sep 6 09:29:51.980246832 (187h21m48.660442411s) [upload=164881]: Done uploading segments...
Sep 6 09:29:51.980277220 (187h21m48.660472789s) [upload=164881]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 8)
Duration: 1m0.174957571s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
1m0.000968106s
FAIL (2025-09-06 09:29:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000981211s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.000874826s target:
https://us1.storj.io/login
FAIL (2025-09-06 09:29:49 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout
Duration: 30.000907488s
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration:
30.00074583s
FAIL (2025-09-06 09:29:46 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 30.000770978s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
54.558297069s
FAIL (2025-09-06 09:29:44 UTC)
orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:05.239202080 (187h19m1.919397649s) [upload=164859]: Starting upload Sep 6 09:27:05.239268026 (187h19m1.919463595s) [upload=164859]: Done starting upload Sep 6 09:27:05.239318563 (187h19m1.919514132s) [upload=164859]: Uploading segments... Sep 6 09:27:05.239355453 (187h19m1.919551022s) [upload=164859]: Got next segment. Inline: true Sep 6 09:27:05.239359901 (187h19m1.91955547s) [upload=164859]: Waiting for error group managing segments... Sep 6 09:27:05.239392204 (187h19m1.919587773s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:05.239396332 (187h19m1.919591901s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:05.239398846 (187h19m1.919594415s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:09.107329409 (187h19m5.787524978s) [upload=164859]: Done uploading segments... Sep 6 09:27:09.107348155 (187h19m5.787543714s) [upload=164859]: Upload finished. err: <nil>
Duration: 54.558326395s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
4 out of 60 is failed (threshold=3) [....................................................X.X.X.X.] duration:
12.679507286s
FAIL (2025-09-06 09:29:43 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 12.679534357s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration:
30.00100606s
FAIL (2025-09-06 09:29:32 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 30.001030206s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
6 out of 60 is failed (threshold=3) [....................................................XXXXXX..] duration:
26.74385219s
FAIL (2025-09-06 09:29:28 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:43.779278335 (187h19m40.459473905s) [upload=164866]: Starting upload Sep 6 09:27:43.779338612 (187h19m40.459534181s) [upload=164866]: Done starting upload Sep 6 09:27:43.779387464 (187h19m40.459583033s) [upload=164866]: Uploading segments... Sep 6 09:27:43.779420899 (187h19m40.459616468s) [upload=164866]: Got next segment. Inline: true Sep 6 09:27:43.779424937 (187h19m40.459620506s) [upload=164866]: Waiting for error group managing segments... Sep 6 09:27:43.779457409 (187h19m40.459652978s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:43.779462638 (187h19m40.459658207s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:43.779465854 (187h19m40.459661413s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:53.418897371 (187h19m50.09909294s) [upload=164866]: Done uploading segments... Sep 6 09:27:53.418915256 (187h19m50.099110825s) [upload=164866]: Upload finished. err: <nil>
Duration: 26.743881597s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
30.446602337s
FAIL (2025-09-06 09:29:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0)
======LOGS======
se=true , address=77.90.31.15:4698 , piece_id=YRVXQUYOUVZKV7IVFUP4T4KTYOYHMGNYLF57T2TEFKHY5GR4K2LA, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:28:50.259470014 (187h20m46.939665583s) [noise=true , address=45.91.116.9:4263 , piece_id=UA3CGEOTIQ3DRQEKQNZDLTCZJ5PZU3NQPGUG5AID4YPLIXSFC5OQ, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:28:50.259550618 (187h20m46.939746177s) [noise=true , address=45.91.116.11:4119 , piece_id=K2DF4VARLQK7C4BEJJM73WPVEFLRVB5GOZQPJUTESRPKTPNOMJSA, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:28:50.259820224 (187h20m46.940015793s) [noise=true , address=77.90.31.12:4589 , piece_id=7OWEEX25MOJ44TBDUSEA4XURYRI2GGJUFITLVC32FXBPF72G353Q, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:28:50.259847506 (187h20m46.940043075s) [noise=true , address=45.91.116.4:4939 , piece_id=IJRWX64S2KXB4LVBEN4JEGJOHD4MWVKOSXQXLZYSV5OOXV5DUQPQ, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:28:50.260045895 (187h20m46.940241454s) [noise=true , address=77.90.31.10:4519 , piece_id=HNOUBWFTHC4CKREIEH2BYIVFFKOG2BKDMT27VTHULGUU7BZQ7FFQ, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:28:50.260249916 (187h20m46.940445485s) [noise=true , address=45.91.116.11:4118 , piece_id=MRWDTIWD7BBG74HFKB33SVQDJXBVLMOYUOOOXN3GLTINSMGT356Q, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:28:50.260463033 (187h20m46.940658592s) [noise=true , address=45.91.116.10:4083 , piece_id=WQDU32VL6FSRR3KGQDHXYDILEAWJMYPPGEYBGYMH2VSRTMXKTKFQ, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:28:50.260671783 (187h20m46.940867362s) [noise=true , address=45.91.116.13:4155 , piece_id=TLXBM7J36GSRQ3Y27VVDTW3ZULJJ6LJPTX43V5OKSOC5NSPXU7MA, seg_pos={0 0}, upload=164880]: Uploading piece...
Sep 6 09:29:11.507184763 (187h21m8.187380352s) [upload=164880]: Next segment err: context deadline exceeded
Sep 6 09:29:11.507514244 (187h21m8.187709833s) [noise=true , address=77.90.31.14:4661 , piece_id=R2F3R35BONGWA5S74VA63U232N7VSDWZBKNHB3T5GRRIMPBNC4EA, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:37248->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37248->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37248->77.90.31.14:4661: use of closed network connection
Sep 6 09:29:11.507737501 (187h21m8.18793307s) [noise=true , address=45.91.116.6:4975 , piece_id=DS5PNZB3CLKSUZ6IJZFYZ6ETGWKHI6BBNS3JAVXZBWCQDO4X74XQ, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: rpc: dial tcp 45.91.116.6:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.507808767 (187h21m8.188004347s) [noise=true , address=77.90.31.12:4589 , piece_id=7OWEEX25MOJ44TBDUSEA4XURYRI2GGJUFITLVC32FXBPF72G353Q, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:55420->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:55420->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55420->77.90.31.12:4589: use of closed network connection
Sep 6 09:29:11.507863382 (187h21m8.188058941s) [noise=true , address=45.91.116.9:4263 , piece_id=UA3CGEOTIQ3DRQEKQNZDLTCZJ5PZU3NQPGUG5AID4YPLIXSFC5OQ, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:49620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508012707 (187h21m8.188208276s) [noise=true , address=45.91.116.12:4225 , piece_id=U6HTSMAJOTZB3H2TDETBNZRQH6W6GJVPVXC7B6QEWNKX2FVJACXQ, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:53622->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508068985 (187h21m8.188264554s) [noise=true , address=45.91.116.12:4226 , piece_id=MEZEFFRZVQEP6RT6UWJTTMQQHHW6TIBLNZ5NSQ5AVGP3SFMY5ZHA, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:58710->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508106347 (187h21m8.188301916s) [noise=true , address=45.91.116.10:4083 , piece_id=WQDU32VL6FSRR3KGQDHXYDILEAWJMYPPGEYBGYMH2VSRTMXKTKFQ, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:50990->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508165159 (187h21m8.188360728s) [noise=true , address=45.91.116.9:4262 , piece_id=TPYITPYXP2KNH6AQEMPBJU4BGOULU3FSK7UBRN7VLWR2MO76DKMQ, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:50218->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508213843 (187h21m8.188409412s) [noise=true , address=77.90.31.15:4699 , piece_id=3G4YSCYD5TFLVLLIDO7RMLEZMKCHWHPESCB7A4JCMJVSL72OV73Q, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:45340->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508245593 (187h21m8.188441162s) [noise=true , address=77.90.31.15:4698 , piece_id=YRVXQUYOUVZKV7IVFUP4T4KTYOYHMGNYLF57T2TEFKHY5GR4K2LA, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:55960->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508253729 (187h21m8.188449298s) [noise=true , address=45.91.116.13:4155 , piece_id=TLXBM7J36GSRQ3Y27VVDTW3ZULJJ6LJPTX43V5OKSOC5NSPXU7MA, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 188.166.225.170:35514->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508291771 (187h21m8.18848734s) [noise=true , address=45.91.116.4:4939 , piece_id=IJRWX64S2KXB4LVBEN4JEGJOHD4MWVKOSXQXLZYSV5OOXV5DUQPQ, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:47952->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508307952 (187h21m8.188503521s) [noise=true , address=45.91.116.11:4119 , piece_id=K2DF4VARLQK7C4BEJJM73WPVEFLRVB5GOZQPJUTESRPKTPNOMJSA, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:41804->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508340545 (187h21m8.188536114s) [noise=true , address=77.90.31.19:4843 , piece_id=WKDBIXO3RBMQSWL6JLILO3TX5ETCCBOE2PUEOBDEPVEWOY23FGGQ, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:43296->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508351436 (187h21m8.188547005s) [noise=true , address=77.90.31.10:4519 , piece_id=HNOUBWFTHC4CKREIEH2BYIVFFKOG2BKDMT27VTHULGUU7BZQ7FFQ, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:34182->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508379620 (187h21m8.188575189s) [noise=true , address=45.91.116.11:4118 , piece_id=MRWDTIWD7BBG74HFKB33SVQDJXBVLMOYUOOOXN3GLTINSMGT356Q, seg_pos={0 0}, upload=164880]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:57460->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:11.508408295 (187h21m8.188603874s) [seg_pos={0 0}, upload=164880]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil>
Duration: 30.446633326s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
1m0.431047843s
FAIL (2025-09-06 09:29:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0)
======LOGS======
ploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 188.166.225.170:40464->45.91.116.6:4979: use of closed network connection; write tcp 188.166.225.170:40464->45.91.116.6:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40464->45.91.116.6:4979: use of closed network connection
Sep 6 09:29:10.469557843 (187h21m7.149753402s) [noise=true , address=77.90.31.17:4796 , piece_id=QAGRUFBWDHNZKNYVMJKVH6PN77BOZ36M6ABJIV6NXQSY24O4O7QA, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: read tcp 188.166.225.170:57462->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:10.469843570 (187h21m7.150039149s) [noise=true , address=77.90.31.15:4725 , piece_id=YXGIZQUPNDKEDIKFDRWIVWGX4MHRJSNH5BXVNQGAD5QYGK7TG7EQ, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, address:77.90.31.15:4725): protocol: write tcp 188.166.225.170:55790->77.90.31.15:4725: use of closed network connection; write tcp 188.166.225.170:55790->77.90.31.15:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55790->77.90.31.15:4725: use of closed network connection
Sep 6 09:29:10.469960493 (187h21m7.150156062s) [noise=true , address=145.79.227.50:4382 , piece_id=OO3V5JAFVIGJUTO66PXWXV2YAT27K6ZCGEAOT47PODA3TSQW2PBA, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1SN5izFGjg4t9qkUVEwF71nNgK2Fjpzo3wuZPJae5qc6QLN8Nf, address:145.79.227.50:4382): protocol: read tcp 188.166.225.170:32784->145.79.227.50:4382: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:10.469995090 (187h21m7.150190649s) [noise=true , address=145.79.227.5:4239 , piece_id=BD2DTDF5HFFNOZNY5COWLN43MWML6OGGGO6IUJHEY4AB47VBS4GA, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:37456->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:37456->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37456->145.79.227.5:4239: use of closed network connection
Sep 6 09:29:10.470186576 (187h21m7.150382165s) [noise=true , address=45.91.116.4:4966 , piece_id=Q4RLRFJP6R7JXU6EXRAXQ4KG5ES2FTMWRTLMUHCAVAIJ65UTPP6Q, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S, address:45.91.116.4:4966): protocol: write tcp 188.166.225.170:50524->45.91.116.4:4966: use of closed network connection; write tcp 188.166.225.170:50524->45.91.116.4:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50524->45.91.116.4:4966: use of closed network connection
Sep 6 09:29:10.470201064 (187h21m7.150396623s) [noise=true , address=145.79.227.47:4288 , piece_id=UPLNECLS5TGVKDT67KSMSY4DFUHGKTGKJPVKKVWJXJA5IGB6UVWA, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:12PuahEyfok8Hk4fBNcdrZQWoQZiCXfNWdbfqPMnbEPmdZQ3R1o, address:145.79.227.47:4288): protocol: write tcp 188.166.225.170:50218->145.79.227.47:4288: use of closed network connection; write tcp 188.166.225.170:50218->145.79.227.47:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50218->145.79.227.47:4288: use of closed network connection
Sep 6 09:29:10.470295555 (187h21m7.150491134s) [noise=true , address=145.79.227.14:4859 , piece_id=VGE66FEQL3DKUTTACUVCGKP5Y2ZSJLJTQPVYCKLBW42OQR774QTA, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:145.79.227.14:4859): protocol: rpc: dial tcp 145.79.227.14:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:10.470341813 (187h21m7.150537382s) [noise=true , address=145.79.227.15:4900 , piece_id=PGCUAOWDPNBZPZQ3KAHCUY6BVCU2RD37KZ6GY4K4N6FZOR3WLK4A, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:12Vmgz1CrhRwUPxEbndWvnGAU25JNYDHgNYfph9qg2yCtH31iFZ, address:145.79.227.15:4900): protocol: write tcp 188.166.225.170:58424->145.79.227.15:4900: use of closed network connection; write tcp 188.166.225.170:58424->145.79.227.15:4900: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58424->145.79.227.15:4900: use of closed network connection
Sep 6 09:29:10.470404172 (187h21m7.150599741s) [noise=true , address=145.79.227.37:4575 , piece_id=HNZZLDYT2SX7IMFY2CQCFHHVO5NHJ7NWMFTDNINSQBPX7Z763T4Q, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:12faeDgwCPWMN4w1RxSPWwPTGNbaKnA87zYX8dhFRgm7F2YLYbV, address:145.79.227.37:4575): protocol: write tcp 188.166.225.170:56588->145.79.227.37:4575: use of closed network connection; write tcp 188.166.225.170:56588->145.79.227.37:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56588->145.79.227.37:4575: use of closed network connection
Sep 6 09:29:10.470425322 (187h21m7.150620892s) [noise=true , address=145.79.227.34:4916 , piece_id=43TBNNGZMFGRRRUNICG3UH4AEWUSMC7XQPHXHXGA6EBFNFAAL5TA, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:145.79.227.34:4916): protocol: write tcp 188.166.225.170:52230->145.79.227.34:4916: use of closed network connection; write tcp 188.166.225.170:52230->145.79.227.34:4916: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52230->145.79.227.34:4916: use of closed network connection
Sep 6 09:29:10.470442957 (187h21m7.150638526s) [noise=true , address=145.79.227.28:4736 , piece_id=GHL26IHLHOQZEJKHA4UIIX42HLSSQWOBFOQC7PV6B3KA6EWH3KBQ, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:126MWCnfveMZSncssS6wf4HGCi4eXvL7FvMD31oYoiR3SEFdGxX, address:145.79.227.28:4736): protocol: write tcp 188.166.225.170:40422->145.79.227.28:4736: use of closed network connection; write tcp 188.166.225.170:40422->145.79.227.28:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40422->145.79.227.28:4736: use of closed network connection
Sep 6 09:29:10.470478284 (187h21m7.150673853s) [noise=true , address=145.79.227.13:4810 , piece_id=RSWR5SDYRAIYNI33AAMU4WR4E7FYEM5CRKFDD3EIHAHXY2CWSQOQ, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:12GTWz2fWHLAApibeARUerDvJ9c4AWYRVAvumRxgJwwqCYYJJfg, address:145.79.227.13:4810): protocol: read tcp 188.166.225.170:44876->145.79.227.13:4810: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:10.470518561 (187h21m7.150714131s) [noise=true , address=145.79.227.2:4141 , piece_id=QUFGH6TWMPDRLE27SSTWB26C5KRPPEZETSFRG4UQRGD5LUFP7Q7A, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:145.79.227.2:4141): protocol: read tcp 188.166.225.170:51592->145.79.227.2:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:10.470553288 (187h21m7.150748857s) [noise=true , address=145.79.227.12:4778 , piece_id=WIHDHEWKYSESGO7H3ZJKYARLNKEOW3Z5XFHHGI6ZODDMMCYAWU3A, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:12JUnoFQa1s1amHUHWiTtQVatCzgMHii1kskRjm9DYjkEuke3Ge, address:145.79.227.12:4778): protocol: read tcp 188.166.225.170:32796->145.79.227.12:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:10.470602973 (187h21m7.150798552s) [noise=true , address=77.90.31.13:4648 , piece_id=ATSNHO5VV267P7KI5M2DEM62AMAFGOMI2V77MXYY5BJANBOWTNBQ, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 188.166.225.170:36018->77.90.31.13:4648: use of closed network connection; write tcp 188.166.225.170:36018->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36018->77.90.31.13:4648: use of closed network connection
Sep 6 09:29:10.470660042 (187h21m7.150855611s) [noise=true , address=145.79.227.17:4992 , piece_id=7GCKZ2PIEATQQ73TOPMQFDIAOY55KI4XAC2VVAD7QXE5O2S7LANA, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:145.79.227.17:4992): protocol: read tcp 188.166.225.170:53654->145.79.227.17:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:10.470702723 (187h21m7.150898292s) [noise=true , address=145.79.227.35:4512 , piece_id=OV6TA4WJ4QVCHU5XM73BLVQSXCPGO26C2RD5FZNUGUNRLBF6WB3Q, seg_pos={0 0}, upload=164875]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:145.79.227.35:4512): protocol: read tcp 188.166.225.170:35470->145.79.227.35:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:29:10.470759582 (187h21m7.150955151s) [seg_pos={0 0}, upload=164875]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil>
Duration: 1m0.431071318s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
3.972788414s
FAIL (2025-09-06 09:29:01 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 3.972817791s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration:
5.241308345s
FAIL (2025-09-06 09:29:00 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 48)
Duration: 5.241409499s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
4 out of 60 is failed (threshold=3) [.....................................................X.X.X.X] duration:
30.000641842s
FAIL (2025-09-06 09:28:57 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.000661831s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
30.00095902s
FAIL (2025-09-06 09:28:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000980059s
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
24.277022888s
FAIL (2025-09-06 09:28:51 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 24.277045111s
edge
authservice
http
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
1m0.000405729s
FAIL (2025-09-06 09:28:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000427591s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
6 out of 60 is failed (threshold=3) [.....................................................XXXXXX.] duration:
19.549776856s
FAIL (2025-09-06 09:28:47 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:43.779278335 (187h19m40.459473905s) [upload=164866]: Starting upload Sep 6 09:27:43.779338612 (187h19m40.459534181s) [upload=164866]: Done starting upload Sep 6 09:27:43.779387464 (187h19m40.459583033s) [upload=164866]: Uploading segments... Sep 6 09:27:43.779420899 (187h19m40.459616468s) [upload=164866]: Got next segment. Inline: true Sep 6 09:27:43.779424937 (187h19m40.459620506s) [upload=164866]: Waiting for error group managing segments... Sep 6 09:27:43.779457409 (187h19m40.459652978s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:43.779462638 (187h19m40.459658207s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:43.779465854 (187h19m40.459661413s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:53.418897371 (187h19m50.09909294s) [upload=164866]: Done uploading segments... Sep 6 09:27:53.418915256 (187h19m50.099110825s) [upload=164866]: Upload finished. err: <nil>
Duration: 19.549869002s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
1m0.176152753s
FAIL (2025-09-06 09:28:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
nil>
Sep 6 09:27:47.372168070 (187h19m44.052363629s) [noise=true , address=89.149.201.147:4005 , piece_id=SPNV4MUKJBELWHZP3RAUV7SZEKVSZZCLTSURPTMTCEEFH2VU5IIQ, seg_pos={0 0}, upload=164867]: Done uploading piece. err: <nil>
Sep 6 09:27:47.395927936 (187h19m44.076123505s) [noise=true , address=89.149.201.147:4014 , piece_id=XDZPADT4ESYLCWKP5GLK7FHPI7IMBOOLR7R4C3DSYBFRKUQ7PVPQ, seg_pos={0 0}, upload=164867]: Done uploading piece. err: <nil>
Sep 6 09:27:47.842570855 (187h19m44.522766424s) [noise=true , address=89.149.201.147:4015 , piece_id=BR7DAWA4HPHAMYTAH2DMASUKVVXYJ5FGKKYVFIQVFS6F2MZ3TIQA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: <nil>
Sep 6 09:28:45.221314035 (187h20m41.901509604s) [noise=true , address=185.167.204.205:4003 , piece_id=Z4N7TXX2FVM3FDJO3DUID7P6677WS3CMG5NWUZTPKP7MZ75DAV3Q, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: write tcp 188.166.225.170:53366->185.167.204.205:4003: use of closed network connection; write tcp 188.166.225.170:53366->185.167.204.205:4003: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53366->185.167.204.205:4003: use of closed network connection
Sep 6 09:28:45.221468792 (187h20m41.901664361s) [noise=true , address=185.167.204.205:4017 , piece_id=TS4EAQMHDPZ2AS64V3AR2XROK3MR3BFUXL5B65KZ35C3VZMBXBSQ, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 188.166.225.170:39384->185.167.204.205:4017: use of closed network connection; write tcp 188.166.225.170:39384->185.167.204.205:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39384->185.167.204.205:4017: use of closed network connection
Sep 6 09:28:45.221512516 (187h20m41.901708085s) [noise=true , address=81.171.14.70:4028 , piece_id=IJ4LXEIBROQA7W576CPNIJDCYBWU5RUJP5RJWSXR446N27XXABZA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12s7cAtvSQB3RQj5km8v3n2gDvwTiBAB7h4cijTE951tPoytDed, address:81.171.14.70:4028): protocol: write tcp 188.166.225.170:55062->81.171.14.70:4028: use of closed network connection; write tcp 188.166.225.170:55062->81.171.14.70:4028: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55062->81.171.14.70:4028: use of closed network connection
Sep 6 09:28:45.221550308 (187h20m41.901745877s) [noise=true , address=81.171.14.70:4048 , piece_id=JKBR3DZWHRXJKV5XER346ZDRC5Y4NW7JMEWDKK2ZIRPZDALQOLPA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:50850->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:50850->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50850->81.171.14.70:4048: use of closed network connection
Sep 6 09:28:45.221582970 (187h20m41.901778539s) [noise=true , address=81.171.14.70:4026 , piece_id=GHJZR4BTQXM46COGCRUWXXGNWQGY4PZCILHJX5IJ2PYIXFYK63KA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:57500->81.171.14.70:4026: use of closed network connection; write tcp 188.166.225.170:57500->81.171.14.70:4026: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57500->81.171.14.70:4026: use of closed network connection
Sep 6 09:28:45.221619591 (187h20m41.90181516s) [noise=true , address=185.167.204.205:4019 , piece_id=NAHVLBNNMZAYAAC6CC7XW3YNXCIB2W4YWTZOFWCFL7HT26LIHV2A, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: write tcp 188.166.225.170:46328->185.167.204.205:4019: use of closed network connection; write tcp 188.166.225.170:46328->185.167.204.205:4019: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46328->185.167.204.205:4019: use of closed network connection
Sep 6 09:28:45.221654126 (187h20m41.901849695s) [noise=true , address=185.167.204.206:4072 , piece_id=JWO53JSRZL6DD7KI2YDISJNGJTRUF76MCBVFVUDZ7A3GO5OBZZEA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:55274->185.167.204.206:4072: use of closed network connection; write tcp 188.166.225.170:55274->185.167.204.206:4072: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55274->185.167.204.206:4072: use of closed network connection
Sep 6 09:28:45.222315261 (187h20m41.90251083s) [noise=true , address=185.167.204.205:4023 , piece_id=OG6VOYCWQKM7O5NN4FNQMYFIMYPWNQ5GLKRTLCIAYA45WUQAG4WA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: read tcp 188.166.225.170:34574->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:45.222354416 (187h20m41.902549985s) [noise=true , address=185.167.204.205:4030 , piece_id=CY47VIYCZKAPKR7Z53S4WN5PCNKGV6PJXAUYEE524T7PZJCYYNRA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: read tcp 188.166.225.170:57272->185.167.204.205:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:45.222388993 (187h20m41.902584562s) [noise=true , address=185.167.204.206:4066 , piece_id=ZHHIWHDXIETA4OTDYBSJV4R4XNQNCBYGCNK53WP3PXIWQ4PS72EA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: read tcp 188.166.225.170:60940->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:45.222421435 (187h20m41.902617004s) [noise=true , address=185.167.204.206:4064 , piece_id=6PC6GXFBVF7FGIOJ3FOSHOZWGY42BAOEMN23TWE6MW33QO7YKKBA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: read tcp 188.166.225.170:41052->185.167.204.206:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:45.222448607 (187h20m41.902644176s) [noise=true , address=185.167.204.205:4033 , piece_id=JBJR7QFILTWVZRNUGXT3YNIU2FC56WRELEGH2BAESB72CETOLBDQ, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: read tcp 188.166.225.170:55064->185.167.204.205:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:45.222482571 (187h20m41.90267813s) [noise=true , address=81.171.14.70:4047 , piece_id=L65O4IQP45IEL7FTN63XMERTFRYBBU2WGIHAUTTLJDMLHZ2VCSYA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m, address:81.171.14.70:4047): protocol: read tcp 188.166.225.170:35946->81.171.14.70:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:45.222514061 (187h20m41.90270962s) [noise=true , address=81.171.14.70:4031 , piece_id=TDZ3OY2U6XMP7LX7YCFM56D5GT2HHH6NCUCMWNT4AMZ3RNH375UQ, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: read tcp 188.166.225.170:54102->81.171.14.70:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:45.222547896 (187h20m41.902743465s) [noise=true , address=185.167.204.206:4058 , piece_id=GVHKVHJ27Z4LKEZKFWVVOQMYZIME5TL7TUBCSZDP77OYBV5DVAQA, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: read tcp 188.166.225.170:53104->185.167.204.206:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:45.222585849 (187h20m41.902781418s) [noise=true , address=185.167.204.206:4063 , piece_id=ATL75SBHCUUSLSF63OWJXZ6IKWPYNP53K7TM6IFZ4GCHPPTAAZKQ, seg_pos={0 0}, upload=164867]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: write tcp 188.166.225.170:37040->185.167.204.206:4063: use of closed network connection; write tcp 188.166.225.170:37040->185.167.204.206:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->185.167.204.206:4063: use of closed network connection
Sep 6 09:28:45.222608903 (187h20m41.902804472s) [seg_pos={0 0}, upload=164867]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 6 09:28:45.394741976 (187h20m42.074937555s) [upload=164867]: Done uploading segments...
Sep 6 09:28:45.394801470 (187h20m42.074997039s) [upload=164867]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 1m0.17616665s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.00032471s target:
https://us1.storj.io/login
FAIL (2025-09-06 09:28:41 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout
Duration: 30.000351832s
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
30.000354947s
FAIL (2025-09-06 09:28:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000374224s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [......................................................X.X.X.] duration:
19.353866541s
FAIL (2025-09-06 09:28:23 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 19.353880748s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
21.05282304s
FAIL (2025-09-06 09:28:20 UTC)
orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:05.239202080 (187h19m1.919397649s) [upload=164859]: Starting upload Sep 6 09:27:05.239268026 (187h19m1.919463595s) [upload=164859]: Done starting upload Sep 6 09:27:05.239318563 (187h19m1.919514132s) [upload=164859]: Uploading segments... Sep 6 09:27:05.239355453 (187h19m1.919551022s) [upload=164859]: Got next segment. Inline: true Sep 6 09:27:05.239359901 (187h19m1.91955547s) [upload=164859]: Waiting for error group managing segments... Sep 6 09:27:05.239392204 (187h19m1.919587773s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:05.239396332 (187h19m1.919591901s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:05.239398846 (187h19m1.919594415s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:09.107329409 (187h19m5.787524978s) [upload=164859]: Done uploading segments... Sep 6 09:27:09.107348155 (187h19m5.787543714s) [upload=164859]: Upload finished. err: <nil>
Duration: 21.052845453s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
30.001313985s
FAIL (2025-09-06 09:28:14 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 6 09:27:48.349015156 (187h19m45.029210725s) [upload=164869]: Starting upload Sep 6 09:27:48.349068678 (187h19m45.029264247s) [upload=164869]: Done starting upload Sep 6 09:27:48.349197334 (187h19m45.029392903s) [upload=164869]: Uploading segments... Sep 6 09:27:48.349382398 (187h19m45.029577967s) [upload=164869]: Got next segment. Inline: false Sep 6 09:27:48.349442182 (187h19m45.029637751s) [upload=164869]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:48.349447071 (187h19m45.02964264s) [upload=164869]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 6 09:28:14.420911026 (187h20m11.101106595s) [upload=164869]: Next segment err: context deadline exceeded Sep 6 09:28:14.421019864 (187h20m11.101215423s) [upload=164869]: Done uploading segments... Sep 6 09:28:14.421033650 (187h20m11.101229219s) [upload=164869]: Upload finished. err: context deadline exceeded
Duration: 30.001333783s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.446532212s
FAIL (2025-09-06 09:28:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0)
======LOGS======
rr: context deadline exceeded
Sep 6 09:28:07.621965459 (187h20m4.302161028s) [noise=true , address=145.79.227.33:4882 , piece_id=JN7RM2HL4OR763FB46MK55BMJZ5DQGPVYBYWLGGWVDPBOA333PBA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:1TU7roqQsKNGvNeqmWdVxMregk9Dqn3YZpfdvK6xp7c3zYvccc, address:145.79.227.33:4882): protocol: write tcp 188.166.225.170:58292->145.79.227.33:4882: use of closed network connection; write tcp 188.166.225.170:58292->145.79.227.33:4882: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58292->145.79.227.33:4882: use of closed network connection
Sep 6 09:28:07.622251547 (187h20m4.302447116s) [noise=true , address=145.79.227.44:4222 , piece_id=GN4UE7PCOEYICQR4PS2TAKGE5FOH667XZJ5VF4JYEKNKALK3TEUQ, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:145.79.227.44:4222): protocol: write tcp 188.166.225.170:43714->145.79.227.44:4222: use of closed network connection; write tcp 188.166.225.170:43714->145.79.227.44:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43714->145.79.227.44:4222: use of closed network connection
Sep 6 09:28:07.622383520 (187h20m4.302579089s) [noise=true , address=145.79.227.4:4188 , piece_id=LYT7TAPBAQGWFGZXF3NT2TVJPF5LT765WCWMQ5DM5UGEEMJ3ETQQ, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:12pLFHQ2Xj6n53w9eLQtMB4A29k3Xh6w716z5s2CGxwPnBoZcgZ, address:145.79.227.4:4188): protocol: write tcp 188.166.225.170:38036->145.79.227.4:4188: use of closed network connection; write tcp 188.166.225.170:38036->145.79.227.4:4188: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38036->145.79.227.4:4188: use of closed network connection
Sep 6 09:28:07.622410902 (187h20m4.302606501s) [noise=true , address=145.79.227.14:4851 , piece_id=AS5I2GXABRIQGUEHOMK37EH3ZV6L5F6YQ4BUS5EOY5A5QZ5BZRVA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:49466->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:49466->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49466->145.79.227.14:4851: use of closed network connection
Sep 6 09:28:07.622505833 (187h20m4.302701422s) [noise=true , address=145.79.227.41:4110 , piece_id=I2FTDER5WQWSINYVICDNYTT3JI34DOCFA2SQVSNTBEANA6X22L2Q, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:145.79.227.41:4110): protocol: read tcp 188.166.225.170:33624->145.79.227.41:4110: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622583041 (187h20m4.30277865s) [noise=true , address=45.91.116.6:4005 , piece_id=HFKYX5ZD43TFNMXC2KZR5M2BRHEZLW6OEQHDANRTUUSQUCHUYGIA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:12W5pHzd3BFgkMPrMhSwTQBSxbSp7khERtcaCbZn1crJobN5zWn, address:45.91.116.6:4005): protocol: read tcp 188.166.225.170:57184->45.91.116.6:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622652063 (187h20m4.302847632s) [noise=true , address=145.79.227.37:4582 , piece_id=SRMMJYNT57ODVAMBFNZEME5GPPURL5OWO74VWK2OZS33SO5VPNXQ, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: read tcp 188.166.225.170:38874->145.79.227.37:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622699734 (187h20m4.302895303s) [noise=true , address=145.79.227.34:4937 , piece_id=V6JPZVHL2K7DKVS6JIKEYIGYQDDFYJI3RU7NSJ3QBPKGEBNUAVUQ, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94, address:145.79.227.34:4937): protocol: write tcp 188.166.225.170:40320->145.79.227.34:4937: use of closed network connection; write tcp 188.166.225.170:40320->145.79.227.34:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40320->145.79.227.34:4937: use of closed network connection
Sep 6 09:28:07.622729762 (187h20m4.302925381s) [noise=true , address=145.79.227.49:4343 , piece_id=X3SDAFFYUNVC53NRYSTFQLY5SN6GX2OOHB6DZEFQA675FQAY24ZQ, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: read tcp 188.166.225.170:47416->145.79.227.49:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622762163 (187h20m4.302957732s) [noise=true , address=145.79.227.47:4288 , piece_id=LZYPOSFPDLOEHBMB4XARUEKCFRMHRBWA2PAU7HAZK7TH2RT2MQJA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:12PuahEyfok8Hk4fBNcdrZQWoQZiCXfNWdbfqPMnbEPmdZQ3R1o, address:145.79.227.47:4288): protocol: write tcp 188.166.225.170:46034->145.79.227.47:4288: use of closed network connection; write tcp 188.166.225.170:46034->145.79.227.47:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46034->145.79.227.47:4288: use of closed network connection
Sep 6 09:28:07.622817068 (187h20m4.303012678s) [noise=true , address=45.91.116.4:4943 , piece_id=IAYQNZGSQLC46TH2L3C3TFSGVQBCHG5CA6WU2ZPKU54KIFR2KEHA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:128QuaUBgwagbp1fCbMDLMncgCth8bd4kaMTnd4ebP1pvQMB1jv, address:45.91.116.4:4943): protocol: read tcp 188.166.225.170:51432->45.91.116.4:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622862566 (187h20m4.303058145s) [noise=true , address=145.79.227.33:4896 , piece_id=FGNGKWTVYRZMII3VKTPMNRVWC6MGQ4VGVU7XXLHMKW22WMLLHKTQ, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:145.79.227.33:4896): protocol: write tcp 188.166.225.170:51742->145.79.227.33:4896: use of closed network connection; write tcp 188.166.225.170:51742->145.79.227.33:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51742->145.79.227.33:4896: use of closed network connection
Sep 6 09:28:07.622907562 (187h20m4.303103161s) [noise=true , address=145.79.227.5:4255 , piece_id=CMYPTO4U4OTX7AH4WHQ2FNKVLGSSNZWSNXCHNYEFO6LBGF5UH6JQ, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:145.79.227.5:4255): protocol: read tcp 188.166.225.170:53192->145.79.227.5:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622917411 (187h20m4.30311298s) [noise=true , address=145.79.227.12:4787 , piece_id=XQJLQFDB74KZYLV7VFS6VHIS7JQTUKJHOFYE4H2FG6PLTTYWALSA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: read tcp 188.166.225.170:40044->145.79.227.12:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622959852 (187h20m4.303155421s) [noise=true , address=77.90.31.18:4830 , piece_id=VKFWXTLDYX7W3TJTCA7OEMV56UUKNRUVAJFPT3ZUDP4JND3IW2BA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: read tcp 188.166.225.170:57488->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622967196 (187h20m4.303162765s) [noise=true , address=77.90.31.17:4776 , piece_id=3WTKQIQOUDIBSEFP4IICKZW2UISTHVRY5ZLFRUCMKX7B7E36OZUA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: read tcp 188.166.225.170:53222->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.622973027 (187h20m4.303168596s) [noise=true , address=145.79.227.24:4607 , piece_id=LGR2SCPECLD5GML7WAHXTHXWBH7JH6AJ322WUCKYB376FB6FTFOA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:1Ex8eUvL1Hqxd5EffckSvBeZDpGrMGhKENyCPvqHybtQf3YQzM, address:145.79.227.24:4607): protocol: write tcp 188.166.225.170:50600->145.79.227.24:4607: use of closed network connection; write tcp 188.166.225.170:50600->145.79.227.24:4607: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50600->145.79.227.24:4607: use of closed network connection
Sep 6 09:28:07.622977115 (187h20m4.303172684s) [noise=true , address=145.79.227.17:4994 , piece_id=LDXYRT4ADRHRD7F7IYUSMEJU4NWUUILMNGI43VUGAB2OL7PB2PMA, seg_pos={0 0}, upload=164860]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: read tcp 188.166.225.170:46444->145.79.227.17:4994: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:28:07.623046368 (187h20m4.303241947s) [seg_pos={0 0}, upload=164860]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil>
Duration: 1m0.4465471s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
3.424124183s
FAIL (2025-09-06 09:28:06 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 3.424179951s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m4.919080104s remote:
"tcp:35.212.10.183:443" target:
https://us1.storj.io/login
FAIL (2025-09-06 09:28:05 UTC)
orbiter warning: Get "https://us1.storj.io/login": EOF
Duration: 1m4.919107406s
core
satellite
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X..X..] duration:
5.674578466s
FAIL (2025-09-06 09:28:04 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 48)
Duration: 5.674603864s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
30.000606018s
FAIL (2025-09-06 09:28:03 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:43.779278335 (187h19m40.459473905s) [upload=164866]: Starting upload Sep 6 09:27:43.779338612 (187h19m40.459534181s) [upload=164866]: Done starting upload Sep 6 09:27:43.779387464 (187h19m40.459583033s) [upload=164866]: Uploading segments... Sep 6 09:27:43.779420899 (187h19m40.459616468s) [upload=164866]: Got next segment. Inline: true Sep 6 09:27:43.779424937 (187h19m40.459620506s) [upload=164866]: Waiting for error group managing segments... Sep 6 09:27:43.779457409 (187h19m40.459652978s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:43.779462638 (187h19m40.459658207s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:43.779465854 (187h19m40.459661413s) [upload=164866]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:53.418897371 (187h19m50.09909294s) [upload=164866]: Done uploading segments... Sep 6 09:27:53.418915256 (187h19m50.099110825s) [upload=164866]: Upload finished. err: <nil>
Duration: 30.000634012s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.......................................................X.X.X] duration:
30.001128177s
FAIL (2025-09-06 09:28:03 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.00115023s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
30.00042888s
FAIL (2025-09-06 09:28:02 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 30.000457595s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
30.000420496s
FAIL (2025-09-06 09:28:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000441076s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.000965331s
FAIL (2025-09-06 09:27:58 UTC)
orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:27:05.239202080 (187h19m1.919397649s) [upload=164859]: Starting upload Sep 6 09:27:05.239268026 (187h19m1.919463595s) [upload=164859]: Done starting upload Sep 6 09:27:05.239318563 (187h19m1.919514132s) [upload=164859]: Uploading segments... Sep 6 09:27:05.239355453 (187h19m1.919551022s) [upload=164859]: Got next segment. Inline: true Sep 6 09:27:05.239359901 (187h19m1.91955547s) [upload=164859]: Waiting for error group managing segments... Sep 6 09:27:05.239392204 (187h19m1.919587773s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:27:05.239396332 (187h19m1.919591901s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:27:05.239398846 (187h19m1.919594415s) [upload=164859]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:27:09.107329409 (187h19m5.787524978s) [upload=164859]: Done uploading segments... Sep 6 09:27:09.107348155 (187h19m5.787543714s) [upload=164859]: Upload finished. err: <nil>
Duration: 1m0.000999054s
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
5.360924234s
FAIL (2025-09-06 09:27:58 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 5.36093773s
edge
authservice
http
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001530425s
FAIL (2025-09-06 09:27:53 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 30.001558929s
edge
authservice
http
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
1m0.175803069s
FAIL (2025-09-06 09:27:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 5)
======LOGS======
HSXQBNH5H6LPLFUQV3PVL7EFJVDBNO7GVIWGE6WRBMKTQA, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799043156 (187h19m41.479238725s) [noise=true , address=185.167.204.206:4045 , piece_id=O3JOKOEZOMKWUKDQ2WGPXPCY3DWLKYLF4SYYAZA3RDAFXYDNXF7Q, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: read tcp 188.166.225.170:55944->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799080127 (187h19m41.479275696s) [noise=true , address=185.167.204.205:4036 , piece_id=7HOCSYOMUDN4EPMAWCXCWVYT7X4ZFUISUB2AW4I34KQOUTIJ5RMQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: read tcp 188.166.225.170:48354->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799113881 (187h19m41.47930945s) [noise=true , address=185.167.204.206:4071 , piece_id=VFKV5AHWS2B4B2WI4OFL6EAC243OIXW3INEJUJBTRZGKFDBJTIMQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1D11dtZNczougJ1KxaPqpiKAREiUZW6P7UenqMkLZzqDEEvjK8, address:185.167.204.206:4071): protocol: read tcp 188.166.225.170:58842->185.167.204.206:4071: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799144129 (187h19m41.479339698s) [noise=true , address=185.167.204.206:4041 , piece_id=3OXFDNGPDIXLGZMSTOVNAEPBHPVZB4MQK5TIPY7KMY5CX63VMWOQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: read tcp 188.166.225.170:50740->185.167.204.206:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799174036 (187h19m41.479369605s) [noise=true , address=185.167.204.206:4048 , piece_id=K7GPIYJDQO2DVZRDELOUSU4MOKIVNODYGTNLMQADU6H4TSU5ACQQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: read tcp 188.166.225.170:56346->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799201739 (187h19m41.479397308s) [noise=true , address=185.167.204.206:4058 , piece_id=22YNX52AU7AMCF36A7HVYE4JMDX73Z4SNNWSSVUHMUWAEAG7M4HA, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: read tcp 188.166.225.170:50182->185.167.204.206:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799233810 (187h19m41.479429379s) [noise=true , address=185.167.204.205:4027 , piece_id=VDEGO625LYY6FPSZFZJQH6S4F3MEGKAG576AL5OT23FZZ6YHSXMQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: read tcp 188.166.225.170:36762->185.167.204.205:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799262455 (187h19m41.479458014s) [noise=true , address=185.167.204.206:4061 , piece_id=CDONTAYTADFRND253T5MPNWGRCVZADOC4CSK56ALURHEWI3AJSKA, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1G18tJ6BsYF3A1sLpRTTGzomqRWsRsj8tbHMJQeR1tZunoTDia, address:185.167.204.206:4061): protocol: read tcp 188.166.225.170:58446->185.167.204.206:4061: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799300288 (187h19m41.479495857s) [noise=true , address=81.171.14.70:4040 , piece_id=QBO65ZZ2R4KRKZFTIUED7NKZJCGCRGHVRZCOUND7QIUIGNTZRJVQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: read tcp 188.166.225.170:37460->81.171.14.70:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799328041 (187h19m41.47952361s) [noise=true , address=185.167.204.206:4064 , piece_id=XBGTTEERRGYRFSLOJFEITXRCVUQU3L4NEMLMNCLDR7LSJONYLWFA, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: read tcp 188.166.225.170:55796->185.167.204.206:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799359952 (187h19m41.479555521s) [noise=true , address=185.167.204.206:4063 , piece_id=6GW3RL7VNO4BHYV2CIHFFJN6TKQ24STFHOH6LHE3VSXXRHXRUALQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: read tcp 188.166.225.170:38256->185.167.204.206:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799386222 (187h19m41.479581792s) [noise=true , address=81.171.14.70:4030 , piece_id=2AFLN3N6RKJBFTKNSQIM3IGXZBELV4BJZVVTX5PFJU4G4VJCYQVA, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:42500->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799426088 (187h19m41.479621657s) [noise=true , address=185.167.204.206:4056 , piece_id=IAKE4FWL7ERBA54K63HWXZCR75JRSW4QWGINW6R7LMJC242KGPQA, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: read tcp 188.166.225.170:37954->185.167.204.206:4056: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799465845 (187h19m41.479661414s) [noise=true , address=185.167.204.205:4022 , piece_id=BWNRORSEP45DZNA4BDIRGPEVCEPBVGTQP4NLYCTTWZXJJKE5JPTQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 188.166.225.170:42184->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799501282 (187h19m41.479696851s) [noise=true , address=185.167.204.206:4046 , piece_id=DYYUL7DN6GNOTPR264RKASNM3Z4TL3VJ6RXOCTZCNXZDTXS7IY5Q, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1YPnmxhn6aS6pLJ7VhwQkHrefetEQwH3ww3sBRyEghNvhzT8X5, address:185.167.204.206:4046): protocol: read tcp 188.166.225.170:49844->185.167.204.206:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799518025 (187h19m41.479713594s) [noise=true , address=81.171.14.70:4037 , piece_id=OXRLAKYTU2WN6HSNE2WO67JB3U7P5LYYNC6FLEUKCYZHT7EGOSYA, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx, address:81.171.14.70:4037): protocol: read tcp 188.166.225.170:50304->81.171.14.70:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799542732 (187h19m41.479738301s) [noise=true , address=185.167.204.206:4072 , piece_id=JY23RKOJIEU4QISFENNVA2KG4RZZHFYFO2EM3EDYPANLJMINPR5Q, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: read tcp 188.166.225.170:37438->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799569053 (187h19m41.479764622s) [noise=true , address=185.167.204.205:4006 , piece_id=T4YC7VKBI7PYSTJEYMNG5Z3YUVQFB2Y342GHVEMASW4PQPQAQQVQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: read tcp 188.166.225.170:58726->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799591214 (187h19m41.479786783s) [noise=true , address=81.171.14.70:4026 , piece_id=KEGSMAJZSOQNABUIGEG5ZCNO2CL75AZUMW2SVRW75G4SS5OWFKSQ, seg_pos={0 0}, upload=164856]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: read tcp 188.166.225.170:60484->81.171.14.70:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:44.799617154 (187h19m41.479812723s) [seg_pos={0 0}, upload=164856]: Done waiting for segment. successful: 5 optimal: 26 errs: <nil>
Sep 6 09:27:44.971845149 (187h19m41.652040778s) [upload=164856]: Done uploading segments...
Sep 6 09:27:44.971876970 (187h19m41.652072539s) [upload=164856]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 5)
Duration: 1m0.175818208s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
33.425685334s
FAIL (2025-09-06 09:27:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0)
======LOGS======
77.90.31.15:4697 , piece_id=ESXNIVFK2ZH2PIF5NXBV47MKHR7JYZPQQJPDJ5RRXLFSF6UWGP6Q, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.284250155 (187h19m28.964445714s) [noise=true , address=45.91.116.9:4263 , piece_id=OIVKPKAME7QG22FLK5CPFWJW2KIHXXVQJW2W3Q2FDOSRCK3UWJIQ, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.284437644 (187h19m28.964633213s) [noise=true , address=77.90.31.14:4663 , piece_id=ZGMWBWZ3QKYWU3XSVOLEDLZTGGUWCBOQIFIPCDMFDTKJRRD7U4RQ, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.284569487 (187h19m28.964765056s) [noise=true , address=77.90.31.17:4771 , piece_id=TQ5TMY6NX5ALAD2VLS573TURBFCG56IGQS4KZB6L3F5AFFXTHTEQ, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.284686430 (187h19m28.964881999s) [noise=true , address=77.90.31.17:4772 , piece_id=7PAGSY6LEZAY4HBKZWXJB7OZH2RVZFZYBGTN4ETOY7N347QRZA3Q, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.284876955 (187h19m28.965072524s) [noise=true , address=45.91.116.11:4117 , piece_id=UC5PRKLVWMGG4ZCOLAEWLO4JLMTXVOZDK3LFAOXXHTKLKGVO6PNQ, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.284898937 (187h19m28.965094506s) [noise=true , address=45.91.116.12:4225 , piece_id=IGETLWG57N22FKAHRP23YKKRQAS6HPGTUS4TDBVJ3MJOLZ3ZSDUA, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.285101805 (187h19m28.965297374s) [noise=true , address=45.91.116.14:4191 , piece_id=V332FBASWGKK7XO3CPAWXVJNFTTJZTBHM2DOTVVRN56WR2SIOCBA, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.285157902 (187h19m28.965353471s) [noise=true , address=45.91.116.12:4226 , piece_id=4B3K5QUOJYOQTO7UHXREM2JGWJYFIFHE345T4THK6RSMM74UQQBA, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.285421346 (187h19m28.965616916s) [noise=true , address=77.90.31.19:4841 , piece_id=XLKXMQM3GZ4K4N4SNNCSUWSURVYKF36ABNKA6IMP5GQXY7OHPOVA, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:32.285742070 (187h19m28.965937639s) [noise=true , address=45.91.116.10:4081 , piece_id=FBFPM3S7NRHROWIQ5XNMORKPGVAN5VT53H2V2NN3QGAAJYKYVWKQ, seg_pos={0 0}, upload=164863]: Uploading piece...
Sep 6 09:27:34.088780455 (187h19m30.768976014s) [upload=164863]: Next segment err: context deadline exceeded
Sep 6 09:27:34.089679545 (187h19m30.769875114s) [noise=true , address=45.91.116.12:4226 , piece_id=4B3K5QUOJYOQTO7UHXREM2JGWJYFIFHE345T4THK6RSMM74UQQBA, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.089735663 (187h19m30.769931232s) [noise=true , address=77.90.31.21:4879 , piece_id=6GP5N76MQNB7SADFUXI6B7FTXTRDRUYAPXF77QS6WQPZIWI553ZQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.089768786 (187h19m30.769964345s) [noise=true , address=77.90.31.17:4772 , piece_id=7PAGSY6LEZAY4HBKZWXJB7OZH2RVZFZYBGTN4ETOY7N347QRZA3Q, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 188.166.225.170:49474->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.089838599 (187h19m30.770034168s) [noise=true , address=45.91.116.14:4191 , piece_id=V332FBASWGKK7XO3CPAWXVJNFTTJZTBHM2DOTVVRN56WR2SIOCBA, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:48918->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090159954 (187h19m30.770355523s) [noise=true , address=77.90.31.19:4842 , piece_id=BP72RAMPQAPOKZZYWE5GPBAV5CQ6QHJHSVLVZG72NAOPOGGK2HFA, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:39314->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090210631 (187h19m30.7704062s) [noise=true , address=45.91.116.5:4299 , piece_id=5R7YH3VZYUCGCWFBFF4EZJL5HOP76H3FIFWZHEMZIXKG24Q4WIWQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47488->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090241089 (187h19m30.770436658s) [noise=true , address=77.90.31.13:4627 , piece_id=Q3BOMHPBCRQI2QSZOXAOAJAQVXS46GN4OUQFGQBUIMUEL7KFO7LQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:60544->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090370567 (187h19m30.770566136s) [noise=true , address=45.91.116.6:4973 , piece_id=W3XUB3IFHXLQQ6LLIQHAXZYC3FRCOVQF2GNNXA2PY3O6OYK25URQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:36928->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090387379 (187h19m30.770582948s) [noise=true , address=77.90.31.15:4697 , piece_id=ESXNIVFK2ZH2PIF5NXBV47MKHR7JYZPQQJPDJ5RRXLFSF6UWGP6Q, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 188.166.225.170:41468->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090420703 (187h19m30.770616263s) [noise=true , address=45.91.116.11:4117 , piece_id=UC5PRKLVWMGG4ZCOLAEWLO4JLMTXVOZDK3LFAOXXHTKLKGVO6PNQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 188.166.225.170:40782->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090440010 (187h19m30.770635579s) [noise=true , address=77.90.31.14:4663 , piece_id=ZGMWBWZ3QKYWU3XSVOLEDLZTGGUWCBOQIFIPCDMFDTKJRRD7U4RQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:38188->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090457504 (187h19m30.770653073s) [noise=true , address=45.91.116.10:4081 , piece_id=FBFPM3S7NRHROWIQ5XNMORKPGVAN5VT53H2V2NN3QGAAJYKYVWKQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:43750->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090479315 (187h19m30.770674885s) [noise=true , address=45.91.116.12:4225 , piece_id=IGETLWG57N22FKAHRP23YKKRQAS6HPGTUS4TDBVJ3MJOLZ3ZSDUA, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:37482->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090485186 (187h19m30.770680756s) [noise=true , address=77.90.31.19:4841 , piece_id=XLKXMQM3GZ4K4N4SNNCSUWSURVYKF36ABNKA6IMP5GQXY7OHPOVA, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 188.166.225.170:42972->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090518891 (187h19m30.77071446s) [noise=true , address=77.90.31.17:4771 , piece_id=TQ5TMY6NX5ALAD2VLS573TURBFCG56IGQS4KZB6L3F5AFFXTHTEQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:44358->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:27:34.090643670 (187h19m30.770839239s) [noise=true , address=45.91.116.9:4263 , piece_id=OIVKPKAME7QG22FLK5CPFWJW2KIHXXVQJW2W3Q2FDOSRCK3UWJIQ, seg_pos={0 0}, upload=164863]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: context deadline exceeded; context deadline exceeded
Sep 6 09:27:34.090675100 (187h19m30.770870669s) [seg_pos={0 0}, upload=164863]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil>
Duration: 33.425708087s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
1m0.001116349s
FAIL (2025-09-06 09:27:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.0011277s
core
uplink
us-select-1
us1-placement-20
|