Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.000875216s FAIL (2025-08-07 21:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 4BQEUPDI5UDRVKUBSTZMOQKGKLTHLTBNWFTORUWZQQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234718776 (11h32m51.824063753s) [noise=true , address=104.36.82.34:4571 , piece_id=APKT66JAXIQNA5UXXM3FCX7JCN5ARKXPDUD6ZJ6WB46MMR2VF3UQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234915192 (11h32m51.824260179s) [noise=true , address=45.91.116.5:4325 , piece_id=P5KGQGDJXUKBS7F7ZK6TIP52U7I2OI74XFJTQSSQRFA2AS4WQDVQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.235686748 (11h32m51.825031735s) [noise=true , address=104.36.82.59:4333 , piece_id=BR6IGKYFNUAH2ZWKQMUQHYGMOL27HFFGN6PEPJOP7WODCYOVUJEQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:53.755589114 (11h32m53.344934101s) [upload=12039]: Next returned nil segment Aug 7 21:12:53.755603641 (11h32m53.344948629s) [upload=12039]: Waiting for error group managing segments... Aug 7 21:12:54.242895872 (11h32m53.832240859s) [noise=true , address=104.36.82.44:4882 , piece_id=M7WROH24EKE2EMGFKDDDS6UOM3E5P4OXLT2QDIBXN2BCYFZCYUMA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.247833398 (11h32m53.837178375s) [noise=true , address=104.36.82.23:4453 , piece_id=OIB2N3NXKLOBJS2UV353TMJMCMK3QM4UGLNKX7P6PUZDT2MRNILQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.248473462 (11h32m53.837818449s) [noise=true , address=104.36.82.16:4300 , piece_id=RVRXMZLRANZJMWSQAF7HTJUULYADEA2I6GPHDTFGKNS6FNWE36VA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.248750062 (11h32m53.83809505s) [noise=true , address=104.36.82.60:4348 , piece_id=KYGK6JIIPPPB3WXEJME7RHT4YU2K5YA6XO2UFS7RVRFC2MK5HL3Q, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.249467144 (11h32m53.838812131s) [noise=true , address=104.36.82.18:4378 , piece_id=ZJJ542SCBSUKTG7QB5PQFTDRPHMFHN32RY6LXNSDLU7X6SCEVBNQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.269168297 (11h32m53.858513304s) [noise=true , address=104.36.82.56:4247 , piece_id=X7EBBRNI7ZUNCR6SYWNS3TTBBQ4Z3BZZYWZNBYFBYRWYTV4TU5VQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.286754543 (11h32m53.87609953s) [noise=true , address=104.36.82.34:4571 , piece_id=APKT66JAXIQNA5UXXM3FCX7JCN5ARKXPDUD6ZJ6WB46MMR2VF3UQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.315127418 (11h32m53.904472395s) [noise=true , address=104.36.82.43:4866 , piece_id=AO5FAMAICZ2WFNV3L75ULE6UDCKGENWEBO47HXEA5ZVZ6KPOBFXQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.339819320 (11h32m53.929164307s) [noise=true , address=104.36.82.55:4195 , piece_id=CKLDMZFNPKVSRXP5A6Z7DIRQOTMLENLOL7XBDWJTMIRDNCFV3ZIA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.352457851 (11h32m53.941802858s) [noise=true , address=104.36.82.59:4333 , piece_id=BR6IGKYFNUAH2ZWKQMUQHYGMOL27HFFGN6PEPJOP7WODCYOVUJEQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.366096104 (11h32m53.955441091s) [noise=true , address=104.36.82.6:4890 , piece_id=DBUDIACR7PEFAFGSOBGEIMWWYDQ4QD4ZWFQ5MNIT2DRDSHWJUE7A, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.366979544 (11h32m53.956324531s) [noise=true , address=104.36.82.62:4427 , piece_id=QTWU24TIM66G6SQXGSU4H22JZTWDPYSZVIZV4FGRILO77SGECVNQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.381141350 (11h32m53.970486347s) [noise=true , address=104.36.82.41:4772 , piece_id=BGC5NOQVRQ4BQEUPDI5UDRVKUBSTZMOQKGKLTHLTBNWFTORUWZQQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.391573819 (11h32m53.980918806s) [noise=true , address=45.91.116.8:4067 , piece_id=GYAFL34K7CTRS3SOTQZFERYUQGQKVOBWUTPJ3E5CFYBZ7ON5RQVA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.391594779 (11h32m53.980939766s) [seg_pos={0 0}, upload=12039]: Segment reached optimal threshold of 14 pieces. Aug 7 21:12:54.391723245 (11h32m53.981068232s) [noise=true , address=45.91.116.7:4034 , piece_id=RDFD2PLXXJ3GRAOR5ZLPKDLRK5IGJTZSBSNVG4A6XFZB75WPJ22Q, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391809851 (11h32m53.981154838s) [noise=true , address=45.91.116.5:4325 , piece_id=P5KGQGDJXUKBS7F7ZK6TIP52U7I2OI74XFJTQSSQRFA2AS4WQDVQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391822284 (11h32m53.981167271s) [noise=true , address=104.36.82.45:4939 , piece_id=UIVN6HS5ZIB44VIN3PBQSMVEEFAYA3DF43KAALY6G6HCFADLZWIQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection; write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection Aug 7 21:12:54.391862171 (11h32m53.981207158s) [noise=true , address=104.36.82.43:4867 , piece_id=ZZCKCBXHV7POWCGCQ7DAJ2QE5A4UYFEERTSCUGVWU7UC76BYZTYA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zENB6bYmb9hqtGTcJ1afKVbcNtbEsGo4dpsSEdh2feqHifmHQ): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391878141 (11h32m53.981223118s) [seg_pos={0 0}, upload=12039]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:12:54.391977522 (11h32m53.981322509s) [upload=12039]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:12:54.391981659 (11h32m53.981326646s) [upload=12039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.855789778 (11h32m54.445134765s) [upload=12039]: Done uploading segments... Aug 7 21:12:54.855808144 (11h32m54.445153121s) [upload=12039]: Upload finished. err: <nil> Aug 7 21:13:45.303666216 (11h33m44.893011253s) [upload=12059]: Starting upload Aug 7 21:13:45.303786125 (11h33m44.893131112s) [upload=12059]: Done starting upload Aug 7 21:13:45.303856099 (11h33m44.893201126s) [upload=12059]: Uploading segments... Aug 7 21:13:45.304260994 (11h33m44.893605981s) [upload=12059]: Got next segment. Inline: false Aug 7 21:13:45.304300049 (11h33m44.893645036s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.304305609 (11h33m44.893650596s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:44.664782364 (11h34m44.254127351s) [upload=12059]: Next segment err: context deadline exceeded Aug 7 21:14:44.664969041 (11h34m44.254314028s) [upload=12059]: Done uploading segments... Aug 7 21:14:44.665004269 (11h34m44.254349256s) [upload=12059]: Upload finished. err: context deadline exceeded Aug 7 21:14:45.557957097 (11h34m45.147302084s) [upload=12072]: Starting upload Aug 7 21:14:45.558041418 (11h34m45.147386395s) [upload=12072]: Done starting upload Aug 7 21:14:45.558153183 (11h34m45.14749817s) [upload=12072]: Uploading segments... Aug 7 21:14:45.558236202 (11h34m45.147581189s) [upload=12072]: Got next segment. Inline: false Aug 7 21:14:45.558263704 (11h34m45.147608691s) [upload=12072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.558268123 (11h34m45.1476131s) [upload=12072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:44.909553125 (11h35m44.498898112s) [upload=12072]: Next segment err: context deadline exceeded Aug 7 21:15:44.909693113 (11h35m44.49903809s) [upload=12072]: Done uploading segments... Aug 7 21:15:44.909705847 (11h35m44.499050834s) [upload=12072]: Upload finished. err: context deadline exceeded Aug 7 21:15:45.780163446 (11h35m45.369508433s) [upload=12088]: Starting upload Aug 7 21:15:45.780230324 (11h35m45.369575311s) [upload=12088]: Done starting upload Aug 7 21:15:45.780343460 (11h35m45.369688497s) [upload=12088]: Uploading segments... Aug 7 21:15:45.780437410 (11h35m45.369782397s) [upload=12088]: Got next segment. Inline: false Aug 7 21:15:45.780481765 (11h35m45.369826772s) [upload=12088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:45.780488508 (11h35m45.369833535s) [upload=12088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:45.135036416 (11h36m44.724381403s) [upload=12088]: Next segment err: context deadline exceeded Aug 7 21:16:45.135160053 (11h36m44.72450504s) [upload=12088]: Done uploading segments... Aug 7 21:16:45.135188378 (11h36m44.724533375s) [upload=12088]: Upload finished. err: context deadline exceeded Aug 7 21:16:46.020697803 (11h36m45.61004279s) [upload=12106]: Starting upload Aug 7 21:16:46.020754431 (11h36m45.610099448s) [upload=12106]: Done starting upload Aug 7 21:16:46.020843461 (11h36m45.610188448s) [upload=12106]: Uploading segments... Aug 7 21:16:46.020994882 (11h36m45.610339869s) [upload=12106]: Got next segment. Inline: false Aug 7 21:16:46.021030459 (11h36m45.610375446s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:46.021035288 (11h36m45.610380276s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:45.371432070 (11h37m44.960777057s) [upload=12106]: Next segment err: context deadline exceeded Aug 7 21:17:45.371471144 (11h37m44.960816142s) [upload=12106]: Done uploading segments... Aug 7 21:17:45.371492135 (11h37m44.960837122s) [upload=12106]: Upload finished. err: context deadline exceeded Duration: 1m0.000886867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................................X.XXXX] duration: 1m0.001347898s FAIL (2025-08-07 21:17:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 78s) [upload=11976]: Starting upload Aug 7 21:09:20.788595918 (11h29m20.377940965s) [upload=11976]: Done starting upload Aug 7 21:09:20.788684948 (11h29m20.378029975s) [upload=11976]: Uploading segments... Aug 7 21:09:20.788747948 (11h29m20.378092935s) [upload=11976]: Got next segment. Inline: true Aug 7 21:09:20.788756344 (11h29m20.378101361s) [upload=11976]: Waiting for error group managing segments... Aug 7 21:09:20.788807752 (11h29m20.378152739s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:20.788813985 (11h29m20.378158972s) [upload=11976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:20.788818584 (11h29m20.378163611s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:21.070630248 (11h29m20.659975235s) [upload=11976]: Done uploading segments... Aug 7 21:09:21.070652190 (11h29m20.659997247s) [upload=11976]: Upload finished. err: <nil> Aug 7 21:09:46.948530536 (11h29m46.537875523s) [upload=11987]: Starting upload Aug 7 21:09:46.948588607 (11h29m46.537933584s) [upload=11987]: Done starting upload Aug 7 21:09:46.948642600 (11h29m46.537987587s) [upload=11987]: Uploading segments... Aug 7 21:09:46.948702786 (11h29m46.538047773s) [upload=11987]: Got next segment. Inline: true Aug 7 21:09:46.948707776 (11h29m46.538052763s) [upload=11987]: Waiting for error group managing segments... Aug 7 21:09:46.948747241 (11h29m46.538092228s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:46.948752340 (11h29m46.538097327s) [upload=11987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:46.948754896 (11h29m46.538099883s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.214261274 (11h29m46.803606261s) [upload=11987]: Done uploading segments... Aug 7 21:09:47.214279799 (11h29m46.803624786s) [upload=11987]: Upload finished. err: <nil> Aug 7 21:10:37.572355395 (11h30m37.161700372s) [upload=11999]: Starting upload Aug 7 21:10:37.572410881 (11h30m37.161755868s) [upload=11999]: Done starting upload Aug 7 21:10:37.572467810 (11h30m37.161812797s) [upload=11999]: Uploading segments... Aug 7 21:10:37.572531982 (11h30m37.161876959s) [upload=11999]: Got next segment. Inline: true Aug 7 21:10:37.572536320 (11h30m37.161881307s) [upload=11999]: Waiting for error group managing segments... Aug 7 21:10:37.572570917 (11h30m37.161915904s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:37.572574955 (11h30m37.161919942s) [upload=11999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:37.572577520 (11h30m37.161922507s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:37.838997375 (11h30m37.428342362s) [upload=11999]: Done uploading segments... Aug 7 21:10:37.839014699 (11h30m37.428359686s) [upload=11999]: Upload finished. err: <nil> Aug 7 21:11:31.258043368 (11h31m30.847388355s) [upload=12015]: Starting upload Aug 7 21:11:31.258101790 (11h31m30.847446777s) [upload=12015]: Done starting upload Aug 7 21:11:31.258163838 (11h31m30.847508825s) [upload=12015]: Uploading segments... Aug 7 21:11:31.258195730 (11h31m30.847540717s) [upload=12015]: Got next segment. Inline: true Aug 7 21:11:31.258199567 (11h31m30.847544565s) [upload=12015]: Waiting for error group managing segments... Aug 7 21:11:31.258240175 (11h31m30.847585162s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.258244853 (11h31m30.847589841s) [upload=12015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.258247358 (11h31m30.847592345s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.535852595 (11h31m31.125197582s) [upload=12015]: Done uploading segments... Aug 7 21:11:31.535871041 (11h31m31.125216028s) [upload=12015]: Upload finished. err: <nil> Aug 7 21:11:57.912240700 (11h31m57.501585687s) [upload=12025]: Starting upload Aug 7 21:11:57.912288532 (11h31m57.501633519s) [upload=12025]: Done starting upload Aug 7 21:11:57.912345871 (11h31m57.501690858s) [upload=12025]: Uploading segments... Aug 7 21:11:57.912374877 (11h31m57.501719874s) [upload=12025]: Got next segment. Inline: true Aug 7 21:11:57.912378794 (11h31m57.501723781s) [upload=12025]: Waiting for error group managing segments... Aug 7 21:11:57.912425353 (11h31m57.501770401s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:57.912431234 (11h31m57.501776232s) [upload=12025]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:57.912435933 (11h31m57.50178095s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:57.261749711 (11h32m56.851094698s) [upload=12025]: Done uploading segments... Aug 7 21:12:57.261780350 (11h32m56.851125337s) [upload=12025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:07.178414027 (11h33m6.767759014s) [upload=12044]: Starting upload Aug 7 21:13:07.178467459 (11h33m6.767812446s) [upload=12044]: Done starting upload Aug 7 21:13:07.178514029 (11h33m6.767859016s) [upload=12044]: Uploading segments... Aug 7 21:13:07.178543715 (11h33m6.767888703s) [upload=12044]: Got next segment. Inline: true Aug 7 21:13:07.178547282 (11h33m6.767892259s) [upload=12044]: Waiting for error group managing segments... Aug 7 21:13:07.178580015 (11h33m6.767925002s) [upload=12044]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:07.178583561 (11h33m6.767928548s) [upload=12044]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:07.178585735 (11h33m6.767930722s) [upload=12044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:07.471379642 (11h33m7.060724629s) [upload=12044]: Done uploading segments... Aug 7 21:13:07.471396245 (11h33m7.060741232s) [upload=12044]: Upload finished. err: <nil> Aug 7 21:13:32.020845488 (11h33m31.610190465s) [upload=12052]: Starting upload Aug 7 21:13:32.020909792 (11h33m31.610254769s) [upload=12052]: Done starting upload Aug 7 21:13:32.020999433 (11h33m31.61034443s) [upload=12052]: Uploading segments... Aug 7 21:13:32.021133279 (11h33m31.610478267s) [upload=12052]: Got next segment. Inline: true Aug 7 21:13:32.021141214 (11h33m31.610486231s) [upload=12052]: Waiting for error group managing segments... Aug 7 21:13:32.021222079 (11h33m31.610567066s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:32.021229403 (11h33m31.61057439s) [upload=12052]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:32.021234362 (11h33m31.610579399s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:31.378507697 (11h34m30.967852684s) [upload=12052]: Done uploading segments... Aug 7 21:14:31.378529929 (11h34m30.967874916s) [upload=12052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:36.610398147 (11h34m36.199743124s) [upload=12069]: Starting upload Aug 7 21:14:36.610597218 (11h34m36.199942205s) [upload=12069]: Done starting upload Aug 7 21:14:36.610671039 (11h34m36.200016016s) [upload=12069]: Uploading segments... Aug 7 21:14:36.610800146 (11h34m36.200145133s) [upload=12069]: Got next segment. Inline: true Aug 7 21:14:36.610805957 (11h34m36.200150944s) [upload=12069]: Waiting for error group managing segments... Aug 7 21:14:36.610869649 (11h34m36.200214636s) [upload=12069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:36.610874759 (11h34m36.200219746s) [upload=12069]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:36.610877584 (11h34m36.200222571s) [upload=12069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:35.947737631 (11h35m35.537082618s) [upload=12069]: Done uploading segments... Aug 7 21:15:35.947756376 (11h35m35.537101353s) [upload=12069]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:45.995643042 (11h35m45.584988019s) [upload=12089]: Starting upload Aug 7 21:15:45.995767078 (11h35m45.585112086s) [upload=12089]: Done starting upload Aug 7 21:15:45.995846110 (11h35m45.585191127s) [upload=12089]: Uploading segments... Aug 7 21:15:45.995939870 (11h35m45.585284917s) [upload=12089]: Got next segment. Inline: true Aug 7 21:15:45.995949598 (11h35m45.585294615s) [upload=12089]: Waiting for error group managing segments... Aug 7 21:15:45.996009222 (11h35m45.585354249s) [upload=12089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:45.996016616 (11h35m45.585361623s) [upload=12089]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:45.996022197 (11h35m45.585367204s) [upload=12089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:45.347388198 (11h36m44.936733185s) [upload=12089]: Done uploading segments... Aug 7 21:16:45.347412374 (11h36m44.936757362s) [upload=12089]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:46.218888633 (11h36m45.80823362s) [upload=12107]: Starting upload Aug 7 21:16:46.218949108 (11h36m45.808294095s) [upload=12107]: Done starting upload Aug 7 21:16:46.219029552 (11h36m45.808374539s) [upload=12107]: Uploading segments... Aug 7 21:16:46.219157317 (11h36m45.808502334s) [upload=12107]: Got next segment. Inline: true Aug 7 21:16:46.219163238 (11h36m45.808508225s) [upload=12107]: Waiting for error group managing segments... Aug 7 21:16:46.219206872 (11h36m45.808551859s) [upload=12107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:46.219212492 (11h36m45.80855747s) [upload=12107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:46.219215178 (11h36m45.808560155s) [upload=12107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:45.569887093 (11h37m45.15923208s) [upload=12107]: Done uploading segments... Aug 7 21:17:45.569902753 (11h37m45.15924774s) [upload=12107]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001363076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.0009601s FAIL (2025-08-07 21:17:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ug 7 21:10:36.218649954 (11h30m35.807994941s) [upload=11998]: Done starting upload Aug 7 21:10:36.218713896 (11h30m35.808058913s) [upload=11998]: Uploading segments... Aug 7 21:10:36.218766106 (11h30m35.808111103s) [upload=11998]: Got next segment. Inline: true Aug 7 21:10:36.218772398 (11h30m35.808117425s) [upload=11998]: Waiting for error group managing segments... Aug 7 21:10:36.218820571 (11h30m35.808165558s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:36.218825550 (11h30m35.808170537s) [upload=11998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:36.218828946 (11h30m35.808173943s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:36.494741534 (11h30m36.084086521s) [upload=11998]: Done uploading segments... Aug 7 21:10:36.494760030 (11h30m36.084105007s) [upload=11998]: Upload finished. err: <nil> Aug 7 21:11:36.918559943 (11h31m36.50790493s) [upload=12018]: Starting upload Aug 7 21:11:36.918615830 (11h31m36.507960817s) [upload=12018]: Done starting upload Aug 7 21:11:36.918660295 (11h31m36.508005282s) [upload=12018]: Uploading segments... Aug 7 21:11:36.918688629 (11h31m36.508033606s) [upload=12018]: Got next segment. Inline: true Aug 7 21:11:36.918691745 (11h31m36.508036732s) [upload=12018]: Waiting for error group managing segments... Aug 7 21:11:36.918721552 (11h31m36.508066539s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:36.918725530 (11h31m36.508070517s) [upload=12018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:36.918727794 (11h31m36.508072781s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:37.211063805 (11h31m36.800408792s) [upload=12018]: Done uploading segments... Aug 7 21:11:37.211087029 (11h31m36.800432026s) [upload=12018]: Upload finished. err: <nil> Aug 7 21:12:15.260577538 (11h32m14.849922525s) [upload=12032]: Starting upload Aug 7 21:12:15.260632023 (11h32m14.84997701s) [upload=12032]: Done starting upload Aug 7 21:12:15.260678571 (11h32m14.850023548s) [upload=12032]: Uploading segments... Aug 7 21:12:15.260706434 (11h32m14.850051421s) [upload=12032]: Got next segment. Inline: true Aug 7 21:12:15.260710011 (11h32m14.850054998s) [upload=12032]: Waiting for error group managing segments... Aug 7 21:12:15.260743897 (11h32m14.850088874s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:15.260747644 (11h32m14.850092631s) [upload=12032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:15.260750389 (11h32m14.850095376s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:15.565323071 (11h32m15.154668058s) [upload=12032]: Done uploading segments... Aug 7 21:12:15.565341296 (11h32m15.154686283s) [upload=12032]: Upload finished. err: <nil> Aug 7 21:12:41.001560374 (11h32m40.590905361s) [upload=12034]: Starting upload Aug 7 21:12:41.001659143 (11h32m40.59100413s) [upload=12034]: Done starting upload Aug 7 21:12:41.001745669 (11h32m40.591090666s) [upload=12034]: Uploading segments... Aug 7 21:12:41.001852704 (11h32m40.591197701s) [upload=12034]: Got next segment. Inline: true Aug 7 21:12:41.001857372 (11h32m40.591202359s) [upload=12034]: Waiting for error group managing segments... Aug 7 21:12:41.001907849 (11h32m40.591252836s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:41.001913169 (11h32m40.591258156s) [upload=12034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:41.001915584 (11h32m40.591260571s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:41.286244505 (11h32m40.875589493s) [upload=12034]: Done uploading segments... Aug 7 21:12:41.286260567 (11h32m40.875605544s) [upload=12034]: Upload finished. err: <nil> Aug 7 21:13:16.624584146 (11h33m16.213929123s) [upload=12047]: Starting upload Aug 7 21:13:16.624659630 (11h33m16.214004607s) [upload=12047]: Done starting upload Aug 7 21:13:16.624710127 (11h33m16.214055114s) [upload=12047]: Uploading segments... Aug 7 21:13:16.624745715 (11h33m16.214090702s) [upload=12047]: Got next segment. Inline: true Aug 7 21:13:16.624750654 (11h33m16.214095641s) [upload=12047]: Waiting for error group managing segments... Aug 7 21:13:16.624782977 (11h33m16.214127964s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:16.624786804 (11h33m16.214131791s) [upload=12047]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:16.624789339 (11h33m16.214134326s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.974391598 (11h33m45.563736596s) [upload=12047]: Done uploading segments... Aug 7 21:13:45.974419923 (11h33m45.56376491s) [upload=12047]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:31.328315035 (11h34m30.917660012s) [upload=12067]: Starting upload Aug 7 21:14:31.328377385 (11h34m30.917722373s) [upload=12067]: Done starting upload Aug 7 21:14:31.328450665 (11h34m30.917795652s) [upload=12067]: Uploading segments... Aug 7 21:14:31.328587126 (11h34m30.917932113s) [upload=12067]: Got next segment. Inline: true Aug 7 21:14:31.328592336 (11h34m30.917937323s) [upload=12067]: Waiting for error group managing segments... Aug 7 21:14:31.328654706 (11h34m30.917999693s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:31.328660026 (11h34m30.918005013s) [upload=12067]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:31.328662631 (11h34m30.918007618s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:00.687780769 (11h35m0.277125756s) [upload=12067]: Done uploading segments... Aug 7 21:15:00.687819252 (11h35m0.27716424s) [upload=12067]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:17.371028003 (11h35m16.96037298s) [upload=12080]: Starting upload Aug 7 21:15:17.371083610 (11h35m16.960428587s) [upload=12080]: Done starting upload Aug 7 21:15:17.371134738 (11h35m16.960479725s) [upload=12080]: Uploading segments... Aug 7 21:15:17.371166228 (11h35m16.960511215s) [upload=12080]: Got next segment. Inline: true Aug 7 21:15:17.371170035 (11h35m16.960515022s) [upload=12080]: Waiting for error group managing segments... Aug 7 21:15:17.371203299 (11h35m16.960548286s) [upload=12080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:17.371207227 (11h35m16.960552214s) [upload=12080]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:17.371209821 (11h35m16.960554809s) [upload=12080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:46.717162709 (11h35m46.306507686s) [upload=12080]: Done uploading segments... Aug 7 21:15:46.717181444 (11h35m46.306526431s) [upload=12080]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:47.625517071 (11h35m47.214862058s) [upload=12090]: Starting upload Aug 7 21:15:47.625581626 (11h35m47.214926613s) [upload=12090]: Done starting upload Aug 7 21:15:47.625639957 (11h35m47.214984944s) [upload=12090]: Uploading segments... Aug 7 21:15:47.625677468 (11h35m47.215022456s) [upload=12090]: Got next segment. Inline: true Aug 7 21:15:47.625681686 (11h35m47.215026673s) [upload=12090]: Waiting for error group managing segments... Aug 7 21:15:47.625720211 (11h35m47.215065198s) [upload=12090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:47.625724749 (11h35m47.215069736s) [upload=12090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:47.625727995 (11h35m47.215072982s) [upload=12090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:16.958194553 (11h36m16.547539541s) [upload=12090]: Done uploading segments... Aug 7 21:16:16.958214662 (11h36m16.547559659s) [upload=12090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:31.885561401 (11h36m31.474906388s) [upload=12101]: Starting upload Aug 7 21:16:31.885625224 (11h36m31.474970211s) [upload=12101]: Done starting upload Aug 7 21:16:31.885680450 (11h36m31.475025437s) [upload=12101]: Uploading segments... Aug 7 21:16:31.885719785 (11h36m31.475064772s) [upload=12101]: Got next segment. Inline: true Aug 7 21:16:31.885724353 (11h36m31.47506934s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:16:31.885761385 (11h36m31.475106372s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:31.885766705 (11h36m31.475111692s) [upload=12101]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:31.885769750 (11h36m31.475114737s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:01.230389747 (11h37m0.819734734s) [upload=12101]: Done uploading segments... Aug 7 21:17:01.230411619 (11h37m0.819756606s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:12.558357565 (11h37m12.147702552s) [upload=12112]: Starting upload Aug 7 21:17:12.558411989 (11h37m12.147756966s) [upload=12112]: Done starting upload Aug 7 21:17:12.558461704 (11h37m12.147806691s) [upload=12112]: Uploading segments... Aug 7 21:17:12.558502081 (11h37m12.147847068s) [upload=12112]: Got next segment. Inline: true Aug 7 21:17:12.558505497 (11h37m12.147850484s) [upload=12112]: Waiting for error group managing segments... Aug 7 21:17:12.558537489 (11h37m12.147882476s) [upload=12112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:12.558541086 (11h37m12.147886073s) [upload=12112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:12.558543370 (11h37m12.147888357s) [upload=12112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:41.917772750 (11h37m41.507117737s) [upload=12112]: Done uploading segments... Aug 7 21:17:41.917811996 (11h37m41.507156993s) [upload=12112]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000980678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....XX.......................X..............................] duration: 1.853358139s FAIL (2025-08-07 21:17:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== pb.BatchRequestItem_ObjectCommit Aug 7 20:43:37.067799677 (11h3m36.657144664s) [upload=11534]: Done uploading segments... Aug 7 20:43:37.067817170 (11h3m36.657162157s) [upload=11534]: Upload finished. err: <nil> Aug 7 20:44:07.448276030 (11h4m7.037621017s) [upload=11539]: Starting upload Aug 7 20:44:07.448326927 (11h4m7.037671904s) [upload=11539]: Done starting upload Aug 7 20:44:07.448378606 (11h4m7.037723593s) [upload=11539]: Uploading segments... Aug 7 20:44:07.448408994 (11h4m7.037753981s) [upload=11539]: Got next segment. Inline: true Aug 7 20:44:07.448414504 (11h4m7.037759491s) [upload=11539]: Waiting for error group managing segments... Aug 7 20:44:07.448448068 (11h4m7.037793055s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:44:07.448451785 (11h4m7.037796762s) [upload=11539]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:44:07.448454070 (11h4m7.037799057s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:44:07.799942249 (11h4m7.389287236s) [upload=11539]: Done uploading segments... Aug 7 20:44:07.799959912 (11h4m7.389304899s) [upload=11539]: Upload finished. err: <nil> Aug 7 20:45:19.440040944 (11h5m19.029385921s) [upload=11564]: Starting upload Aug 7 20:45:19.440090409 (11h5m19.029435386s) [upload=11564]: Done starting upload Aug 7 20:45:19.440146045 (11h5m19.029491032s) [upload=11564]: Uploading segments... Aug 7 20:45:19.440190359 (11h5m19.029535346s) [upload=11564]: Got next segment. Inline: true Aug 7 20:45:19.440194267 (11h5m19.029539254s) [upload=11564]: Waiting for error group managing segments... Aug 7 20:45:19.440226098 (11h5m19.029571085s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:45:19.440230026 (11h5m19.029575013s) [upload=11564]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:45:19.440232290 (11h5m19.029577277s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:45:19.858443364 (11h5m19.447788352s) [upload=11564]: Done uploading segments... Aug 7 20:45:19.858467080 (11h5m19.447812067s) [upload=11564]: Upload finished. err: <nil> Aug 7 20:46:00.840349205 (11h6m0.429694192s) [upload=11575]: Starting upload Aug 7 20:46:00.840387950 (11h6m0.429732927s) [upload=11575]: Done starting upload Aug 7 20:46:00.840431102 (11h6m0.429776079s) [upload=11575]: Uploading segments... Aug 7 20:46:00.840473594 (11h6m0.429818581s) [upload=11575]: Got next segment. Inline: true Aug 7 20:46:00.840477311 (11h6m0.429822298s) [upload=11575]: Waiting for error group managing segments... Aug 7 20:46:00.840510584 (11h6m0.429855571s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:00.840514551 (11h6m0.429859539s) [upload=11575]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:00.840516746 (11h6m0.429861733s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:01.149526394 (11h6m0.738871381s) [upload=11575]: Done uploading segments... Aug 7 20:46:01.149544077 (11h6m0.738889064s) [upload=11575]: Upload finished. err: <nil> Aug 7 20:46:32.531571609 (11h6m32.120916586s) [upload=11585]: Starting upload Aug 7 20:46:32.531627656 (11h6m32.120972643s) [upload=11585]: Done starting upload Aug 7 20:46:32.531679334 (11h6m32.121024311s) [upload=11585]: Uploading segments... Aug 7 20:46:32.531716416 (11h6m32.121061403s) [upload=11585]: Got next segment. Inline: true Aug 7 20:46:32.531720834 (11h6m32.121065821s) [upload=11585]: Waiting for error group managing segments... Aug 7 20:46:32.531758977 (11h6m32.121103964s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:32.531762984 (11h6m32.121107961s) [upload=11585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:32.531766161 (11h6m32.121111148s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:32.823368074 (11h6m32.412713051s) [upload=11585]: Done uploading segments... Aug 7 20:46:32.823385747 (11h6m32.412730724s) [upload=11585]: Upload finished. err: <nil> Aug 7 20:46:53.991475826 (11h6m53.580820813s) [upload=11590]: Starting upload Aug 7 20:46:53.991528908 (11h6m53.580873895s) [upload=11590]: Done starting upload Aug 7 20:46:53.991580286 (11h6m53.580925273s) [upload=11590]: Uploading segments... Aug 7 20:46:53.991609822 (11h6m53.58095481s) [upload=11590]: Got next segment. Inline: true Aug 7 20:46:53.991613780 (11h6m53.580958767s) [upload=11590]: Waiting for error group managing segments... Aug 7 20:46:53.991645681 (11h6m53.580990668s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:53.991649578 (11h6m53.580994565s) [upload=11590]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:53.991651822 (11h6m53.580996809s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:54.273031788 (11h6m53.862376835s) [upload=11590]: Done uploading segments... Aug 7 20:46:54.273056015 (11h6m53.862401032s) [upload=11590]: Upload finished. err: <nil> Aug 7 20:48:05.936480351 (11h8m5.525825338s) [upload=11610]: Starting upload Aug 7 20:48:05.936525999 (11h8m5.525870986s) [upload=11610]: Done starting upload Aug 7 20:48:05.936586324 (11h8m5.525931311s) [upload=11610]: Uploading segments... Aug 7 20:48:05.936615510 (11h8m5.525960487s) [upload=11610]: Got next segment. Inline: true Aug 7 20:48:05.936618666 (11h8m5.525963653s) [upload=11610]: Waiting for error group managing segments... Aug 7 20:48:05.936688099 (11h8m5.526033086s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:48:05.936692588 (11h8m5.526037575s) [upload=11610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:48:05.936694892 (11h8m5.526039879s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:48:06.223792648 (11h8m5.813137635s) [upload=11610]: Done uploading segments... Aug 7 20:48:06.223807756 (11h8m5.813152743s) [upload=11610]: Upload finished. err: <nil> Aug 7 20:49:00.370365369 (11h8m59.959710356s) [upload=11624]: Starting upload Aug 7 20:49:00.370414633 (11h8m59.95975962s) [upload=11624]: Done starting upload Aug 7 20:49:00.370467765 (11h8m59.959812752s) [upload=11624]: Uploading segments... Aug 7 20:49:00.370501249 (11h8m59.959846236s) [upload=11624]: Got next segment. Inline: true Aug 7 20:49:00.370504986 (11h8m59.959849973s) [upload=11624]: Waiting for error group managing segments... Aug 7 20:49:00.370538900 (11h8m59.959883887s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:00.370542898 (11h8m59.959887885s) [upload=11624]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:00.370545453 (11h8m59.95989043s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:00.736326238 (11h9m0.325671225s) [upload=11624]: Done uploading segments... Aug 7 20:49:00.736343821 (11h9m0.325688808s) [upload=11624]: Upload finished. err: <nil> Aug 7 20:49:43.904464651 (11h9m43.493809639s) [upload=11637]: Starting upload Aug 7 20:49:43.904540737 (11h9m43.493885724s) [upload=11637]: Done starting upload Aug 7 20:49:43.904599109 (11h9m43.493944096s) [upload=11637]: Uploading segments... Aug 7 20:49:43.904635608 (11h9m43.493980595s) [upload=11637]: Got next segment. Inline: true Aug 7 20:49:43.904639476 (11h9m43.493984463s) [upload=11637]: Waiting for error group managing segments... Aug 7 20:49:43.904675195 (11h9m43.494020182s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:43.904679252 (11h9m43.494024239s) [upload=11637]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:43.904683430 (11h9m43.494028417s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:44.269042214 (11h9m43.858387201s) [upload=11637]: Done uploading segments... Aug 7 20:49:44.269058595 (11h9m43.858403582s) [upload=11637]: Upload finished. err: <nil> Aug 7 20:50:51.186360319 (11h10m50.775705306s) [upload=11656]: Starting upload Aug 7 20:50:51.186464308 (11h10m50.775809295s) [upload=11656]: Done starting upload Aug 7 20:50:51.186527319 (11h10m50.775872306s) [upload=11656]: Uploading segments... Aug 7 20:50:51.186558759 (11h10m50.775903746s) [upload=11656]: Got next segment. Inline: true Aug 7 20:50:51.186562295 (11h10m50.775907282s) [upload=11656]: Waiting for error group managing segments... Aug 7 20:50:51.186595108 (11h10m50.775940095s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:50:51.186599105 (11h10m50.775944082s) [upload=11656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:50:51.186601591 (11h10m50.775946578s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:50:51.466357188 (11h10m51.055702175s) [upload=11656]: Done uploading segments... Aug 7 20:50:51.466375784 (11h10m51.055720781s) [upload=11656]: Upload finished. err: <nil> Aug 7 20:51:54.183767656 (11h11m53.773112643s) [upload=11674]: Starting upload Aug 7 20:51:54.183810869 (11h11m53.773155846s) [upload=11674]: Done starting upload Aug 7 20:51:54.183850504 (11h11m53.773195481s) [upload=11674]: Uploading segments... Aug 7 20:51:54.183877035 (11h11m53.773222012s) [upload=11674]: Got next segment. Inline: true Aug 7 20:51:54.183880662 (11h11m53.773225639s) [upload=11674]: Waiting for error group managing segments... Aug 7 20:51:54.183910529 (11h11m53.773255516s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:51:54.183914426 (11h11m53.773259413s) [upload=11674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:51:54.183916961 (11h11m53.773261938s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:51:54.662596438 (11h11m54.251941425s) [upload=11674]: Done uploading segments... Aug 7 20:51:54.662619702 (11h11m54.251964689s) [upload=11674]: Upload finished. err: <nil> Duration:* 1.853387156s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.000875494s FAIL (2025-08-07 21:17:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000895912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................................................X..XXX] duration: 30.000968157s FAIL (2025-08-07 21:17:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000977424s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................................X..XXXXX] duration: 30.001444076s FAIL (2025-08-07 21:17:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== M6PMA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574518488 (11h33m16.163863465s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574831648 (11h33m16.164176635s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:18.308274980 (11h33m17.897619967s) [upload=12046]: Next returned nil segment Aug 7 21:13:18.308295720 (11h33m17.897640707s) [upload=12046]: Waiting for error group managing segments... Aug 7 21:13:18.656166252 (11h33m18.245511229s) [noise=true , address=77.90.31.13:4626 , piece_id=S4OKGN6SKZ7GLYKCT3S4QSQFK75FX3KAXGQAOORJRVDJ7HV5UTWQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.668187531 (11h33m18.257532518s) [noise=true , address=104.36.82.6:4873 , piece_id=VLAKUO36OHVGPY3Q723O5OUKURJ2SWREZC3SABUX2J7O6MBM7Y2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.721769396 (11h33m18.311114384s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.741305974 (11h33m18.330650951s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.780529499 (11h33m18.369874486s) [noise=true , address=77.90.31.11:4553 , piece_id=4QVTWAAY3JK3EQJIHK7ANW6E4UYUI3KXCHZOWRK75JHZZRPZ6ENQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.865633524 (11h33m18.454978511s) [noise=true , address=45.91.116.11:4119 , piece_id=6PDH442NIXRNQCIFX2Y4PNKU5C73T2H6KNHKGMIW2UGCTC5NX7ZA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880648330 (11h33m18.469993317s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880718855 (11h33m18.470063842s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.935042530 (11h33m18.524387517s) [noise=true , address=45.91.116.5:4298 , piece_id=Y4FCHPKGJXDGWLH2FT6M5PJV5RA4UOYFQ3SHTWFJN4WO3FTFFG2A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.969979703 (11h33m18.55932468s) [noise=true , address=45.91.116.5:4299 , piece_id=5YQ5OJAPBA3N2KMYSVWQCNOTH7QPX2WCHHUXTLNSVNH2P7MZEJ3A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.972928735 (11h33m18.562273722s) [noise=true , address=45.91.116.12:4226 , piece_id=5QFSWOPQLZO2KEP4J27RDULUD4VCQSIE6BZ6BO3MFVFAP67OMW7A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.047677267 (11h33m18.637022254s) [noise=true , address=104.36.82.15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082384121 (11h33m18.671729108s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082409660 (11h33m18.671754647s) [seg_pos={0 0}, upload=12046]: Segment reached optimal threshold of 13 pieces. Aug 7 21:13:19.082628518 (11h33m18.671973495s) [noise=true , address=77.90.31.14:4661 , piece_id=P4DQFG2ACQKL52JQZFCTIFCUHFPLDIYEOOG4OPBJY4MIXBVKCYDQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Aug 7 21:13:19.082795017 (11h33m18.672139994s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: read tcp 188.166.225.170:56758->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082822870 (11h33m18.672167857s) [noise=true , address=104.36.82.16:4294 , piece_id=QN2G4ZPHQ2VBVMEJFPY3OLOGLHZRKEFC4SXOXSHFBQMD47QMAM7Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:47752->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082839501 (11h33m18.672184478s) [seg_pos={0 0}, upload=12046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:13:19.082910688 (11h33m18.672255675s) [upload=12046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:19.082914756 (11h33m18.672259743s) [upload=12046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:19.387715025 (11h33m18.977060012s) [upload=12046]: Done uploading segments... Aug 7 21:13:19.387739612 (11h33m18.977084599s) [upload=12046]: Upload finished. err: <nil> Aug 7 21:13:45.069592425 (11h33m44.658937412s) [upload=12058]: Starting upload Aug 7 21:13:45.069646459 (11h33m44.658991446s) [upload=12058]: Done starting upload Aug 7 21:13:45.069729608 (11h33m44.659074595s) [upload=12058]: Uploading segments... Aug 7 21:13:45.069894263 (11h33m44.65923925s) [upload=12058]: Got next segment. Inline: false Aug 7 21:13:45.069927166 (11h33m44.659272153s) [upload=12058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.069931384 (11h33m44.659276371s) [upload=12058]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:14.428211569 (11h34m14.017556556s) [upload=12058]: Next segment err: context deadline exceeded Aug 7 21:14:14.428287775 (11h34m14.017632762s) [upload=12058]: Done uploading segments... Aug 7 21:14:14.428338853 (11h34m14.01768384s) [upload=12058]: Upload finished. err: context deadline exceeded Aug 7 21:14:49.077304183 (11h34m48.66664917s) [upload=12074]: Starting upload Aug 7 21:14:49.077411358 (11h34m48.666756365s) [upload=12074]: Done starting upload Aug 7 21:14:49.077536547 (11h34m48.666881534s) [upload=12074]: Uploading segments... Aug 7 21:14:49.077698226 (11h34m48.667043214s) [upload=12074]: Got next segment. Inline: false Aug 7 21:14:49.077743253 (11h34m48.66708824s) [upload=12074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:49.077748543 (11h34m48.66709353s) [upload=12074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:18.426190970 (11h35m18.015535997s) [upload=12074]: Next segment err: context deadline exceeded Aug 7 21:15:18.426350985 (11h35m18.015695972s) [upload=12074]: Done uploading segments... Aug 7 21:15:18.426379730 (11h35m18.015724707s) [upload=12074]: Upload finished. err: context deadline exceeded Aug 7 21:15:36.581505585 (11h35m36.170850572s) [upload=12086]: Starting upload Aug 7 21:15:36.581560250 (11h35m36.170905227s) [upload=12086]: Done starting upload Aug 7 21:15:36.581682133 (11h35m36.17102711s) [upload=12086]: Uploading segments... Aug 7 21:15:36.581769470 (11h35m36.171114457s) [upload=12086]: Got next segment. Inline: false Aug 7 21:15:36.581824996 (11h35m36.171169983s) [upload=12086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:36.581829786 (11h35m36.171174773s) [upload=12086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:05.932290909 (11h36m5.521635897s) [upload=12086]: Next segment err: context deadline exceeded Aug 7 21:16:05.932368198 (11h36m5.521713185s) [upload=12086]: Done uploading segments... Aug 7 21:16:05.932396572 (11h36m5.521741559s) [upload=12086]: Upload finished. err: context deadline exceeded Aug 7 21:16:06.820264050 (11h36m6.409609037s) [upload=12095]: Starting upload Aug 7 21:16:06.820316621 (11h36m6.409661608s) [upload=12095]: Done starting upload Aug 7 21:16:06.820429747 (11h36m6.409774734s) [upload=12095]: Uploading segments... Aug 7 21:16:06.820494201 (11h36m6.409839188s) [upload=12095]: Got next segment. Inline: false Aug 7 21:16:06.820519059 (11h36m6.409864046s) [upload=12095]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:06.820524339 (11h36m6.409869326s) [upload=12095]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:36.174935298 (11h36m35.764280285s) [upload=12095]: Next segment err: context deadline exceeded Aug 7 21:16:36.175145370 (11h36m35.764490357s) [upload=12095]: Done uploading segments... Aug 7 21:16:36.175169056 (11h36m35.764514043s) [upload=12095]: Upload finished. err: context deadline exceeded Aug 7 21:16:45.112542660 (11h36m44.701887647s) [upload=12105]: Starting upload Aug 7 21:16:45.112631901 (11h36m44.701976888s) [upload=12105]: Done starting upload Aug 7 21:16:45.112702636 (11h36m44.702047623s) [upload=12105]: Uploading segments... Aug 7 21:16:45.112971772 (11h36m44.702316759s) [upload=12105]: Got next segment. Inline: false Aug 7 21:16:45.113031576 (11h36m44.702376573s) [upload=12105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:45.113037667 (11h36m44.702382694s) [upload=12105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:14.466436969 (11h37m14.055781956s) [upload=12105]: Next segment err: context deadline exceeded Aug 7 21:17:14.466505420 (11h37m14.055850407s) [upload=12105]: Done uploading segments... Aug 7 21:17:14.466521220 (11h37m14.055866207s) [upload=12105]: Upload finished. err: context deadline exceeded Duration: 30.001465697s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.001333344s FAIL (2025-08-07 21:17:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00135224s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.000889584s FAIL (2025-08-07 21:17:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000909052s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.000828684s FAIL (2025-08-07 21:17:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=11995]: Starting upload Aug 7 21:10:17.433520777 (11h30m17.022865764s) [upload=11995]: Done starting upload Aug 7 21:10:17.433584859 (11h30m17.022929846s) [upload=11995]: Uploading segments... Aug 7 21:10:17.433692035 (11h30m17.023037012s) [upload=11995]: Got next segment. Inline: true Aug 7 21:10:17.433696885 (11h30m17.023041872s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:10:17.433744876 (11h30m17.023089853s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:17.433749755 (11h30m17.023094742s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:17.433754204 (11h30m17.023099191s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:17.710858662 (11h30m17.300203709s) [upload=11995]: Done uploading segments... Aug 7 21:10:17.710881997 (11h30m17.300227024s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:10:36.218517401 (11h30m35.807862378s) [upload=11998]: Starting upload Aug 7 21:10:36.218649954 (11h30m35.807994941s) [upload=11998]: Done starting upload Aug 7 21:10:36.218713896 (11h30m35.808058913s) [upload=11998]: Uploading segments... Aug 7 21:10:36.218766106 (11h30m35.808111103s) [upload=11998]: Got next segment. Inline: true Aug 7 21:10:36.218772398 (11h30m35.808117425s) [upload=11998]: Waiting for error group managing segments... Aug 7 21:10:36.218820571 (11h30m35.808165558s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:36.218825550 (11h30m35.808170537s) [upload=11998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:36.218828946 (11h30m35.808173943s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:36.494741534 (11h30m36.084086521s) [upload=11998]: Done uploading segments... Aug 7 21:10:36.494760030 (11h30m36.084105007s) [upload=11998]: Upload finished. err: <nil> Aug 7 21:11:36.918559943 (11h31m36.50790493s) [upload=12018]: Starting upload Aug 7 21:11:36.918615830 (11h31m36.507960817s) [upload=12018]: Done starting upload Aug 7 21:11:36.918660295 (11h31m36.508005282s) [upload=12018]: Uploading segments... Aug 7 21:11:36.918688629 (11h31m36.508033606s) [upload=12018]: Got next segment. Inline: true Aug 7 21:11:36.918691745 (11h31m36.508036732s) [upload=12018]: Waiting for error group managing segments... Aug 7 21:11:36.918721552 (11h31m36.508066539s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:36.918725530 (11h31m36.508070517s) [upload=12018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:36.918727794 (11h31m36.508072781s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:37.211063805 (11h31m36.800408792s) [upload=12018]: Done uploading segments... Aug 7 21:11:37.211087029 (11h31m36.800432026s) [upload=12018]: Upload finished. err: <nil> Aug 7 21:12:15.260577538 (11h32m14.849922525s) [upload=12032]: Starting upload Aug 7 21:12:15.260632023 (11h32m14.84997701s) [upload=12032]: Done starting upload Aug 7 21:12:15.260678571 (11h32m14.850023548s) [upload=12032]: Uploading segments... Aug 7 21:12:15.260706434 (11h32m14.850051421s) [upload=12032]: Got next segment. Inline: true Aug 7 21:12:15.260710011 (11h32m14.850054998s) [upload=12032]: Waiting for error group managing segments... Aug 7 21:12:15.260743897 (11h32m14.850088874s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:15.260747644 (11h32m14.850092631s) [upload=12032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:15.260750389 (11h32m14.850095376s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:15.565323071 (11h32m15.154668058s) [upload=12032]: Done uploading segments... Aug 7 21:12:15.565341296 (11h32m15.154686283s) [upload=12032]: Upload finished. err: <nil> Aug 7 21:12:41.001560374 (11h32m40.590905361s) [upload=12034]: Starting upload Aug 7 21:12:41.001659143 (11h32m40.59100413s) [upload=12034]: Done starting upload Aug 7 21:12:41.001745669 (11h32m40.591090666s) [upload=12034]: Uploading segments... Aug 7 21:12:41.001852704 (11h32m40.591197701s) [upload=12034]: Got next segment. Inline: true Aug 7 21:12:41.001857372 (11h32m40.591202359s) [upload=12034]: Waiting for error group managing segments... Aug 7 21:12:41.001907849 (11h32m40.591252836s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:41.001913169 (11h32m40.591258156s) [upload=12034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:41.001915584 (11h32m40.591260571s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:41.286244505 (11h32m40.875589493s) [upload=12034]: Done uploading segments... Aug 7 21:12:41.286260567 (11h32m40.875605544s) [upload=12034]: Upload finished. err: <nil> Aug 7 21:13:16.624584146 (11h33m16.213929123s) [upload=12047]: Starting upload Aug 7 21:13:16.624659630 (11h33m16.214004607s) [upload=12047]: Done starting upload Aug 7 21:13:16.624710127 (11h33m16.214055114s) [upload=12047]: Uploading segments... Aug 7 21:13:16.624745715 (11h33m16.214090702s) [upload=12047]: Got next segment. Inline: true Aug 7 21:13:16.624750654 (11h33m16.214095641s) [upload=12047]: Waiting for error group managing segments... Aug 7 21:13:16.624782977 (11h33m16.214127964s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:16.624786804 (11h33m16.214131791s) [upload=12047]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:16.624789339 (11h33m16.214134326s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.974391598 (11h33m45.563736596s) [upload=12047]: Done uploading segments... Aug 7 21:13:45.974419923 (11h33m45.56376491s) [upload=12047]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:31.328315035 (11h34m30.917660012s) [upload=12067]: Starting upload Aug 7 21:14:31.328377385 (11h34m30.917722373s) [upload=12067]: Done starting upload Aug 7 21:14:31.328450665 (11h34m30.917795652s) [upload=12067]: Uploading segments... Aug 7 21:14:31.328587126 (11h34m30.917932113s) [upload=12067]: Got next segment. Inline: true Aug 7 21:14:31.328592336 (11h34m30.917937323s) [upload=12067]: Waiting for error group managing segments... Aug 7 21:14:31.328654706 (11h34m30.917999693s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:31.328660026 (11h34m30.918005013s) [upload=12067]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:31.328662631 (11h34m30.918007618s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:00.687780769 (11h35m0.277125756s) [upload=12067]: Done uploading segments... Aug 7 21:15:00.687819252 (11h35m0.27716424s) [upload=12067]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:17.371028003 (11h35m16.96037298s) [upload=12080]: Starting upload Aug 7 21:15:17.371083610 (11h35m16.960428587s) [upload=12080]: Done starting upload Aug 7 21:15:17.371134738 (11h35m16.960479725s) [upload=12080]: Uploading segments... Aug 7 21:15:17.371166228 (11h35m16.960511215s) [upload=12080]: Got next segment. Inline: true Aug 7 21:15:17.371170035 (11h35m16.960515022s) [upload=12080]: Waiting for error group managing segments... Aug 7 21:15:17.371203299 (11h35m16.960548286s) [upload=12080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:17.371207227 (11h35m16.960552214s) [upload=12080]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:17.371209821 (11h35m16.960554809s) [upload=12080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:46.717162709 (11h35m46.306507686s) [upload=12080]: Done uploading segments... Aug 7 21:15:46.717181444 (11h35m46.306526431s) [upload=12080]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:47.625517071 (11h35m47.214862058s) [upload=12090]: Starting upload Aug 7 21:15:47.625581626 (11h35m47.214926613s) [upload=12090]: Done starting upload Aug 7 21:15:47.625639957 (11h35m47.214984944s) [upload=12090]: Uploading segments... Aug 7 21:15:47.625677468 (11h35m47.215022456s) [upload=12090]: Got next segment. Inline: true Aug 7 21:15:47.625681686 (11h35m47.215026673s) [upload=12090]: Waiting for error group managing segments... Aug 7 21:15:47.625720211 (11h35m47.215065198s) [upload=12090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:47.625724749 (11h35m47.215069736s) [upload=12090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:47.625727995 (11h35m47.215072982s) [upload=12090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:16.958194553 (11h36m16.547539541s) [upload=12090]: Done uploading segments... Aug 7 21:16:16.958214662 (11h36m16.547559659s) [upload=12090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:31.885561401 (11h36m31.474906388s) [upload=12101]: Starting upload Aug 7 21:16:31.885625224 (11h36m31.474970211s) [upload=12101]: Done starting upload Aug 7 21:16:31.885680450 (11h36m31.475025437s) [upload=12101]: Uploading segments... Aug 7 21:16:31.885719785 (11h36m31.475064772s) [upload=12101]: Got next segment. Inline: true Aug 7 21:16:31.885724353 (11h36m31.47506934s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:16:31.885761385 (11h36m31.475106372s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:31.885766705 (11h36m31.475111692s) [upload=12101]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:31.885769750 (11h36m31.475114737s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:01.230389747 (11h37m0.819734734s) [upload=12101]: Done uploading segments... Aug 7 21:17:01.230411619 (11h37m0.819756606s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000867688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX...........X.......................X.....] duration: 6.770529533s FAIL (2025-08-07 21:16:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 7106407 (11h31m46.936451394s) [upload=12022]: Uploading segments... Aug 7 21:11:47.347198184 (11h31m46.936543161s) [upload=12022]: Got next segment. Inline: false Aug 7 21:11:47.347234122 (11h31m46.936579109s) [upload=12022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:47.347237969 (11h31m46.936582956s) [upload=12022]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:11:47.802049075 (11h31m47.391394053s) [seg_pos={0 0}, upload=12022]: Begin upload segment... Aug 7 21:11:47.802118007 (11h31m47.391462994s) [seg_pos={0 0}, upload=12022]: Done begin upload segment. Aug 7 21:11:47.802142414 (11h31m47.391487401s) [noise=true , address=104.36.82.22:4541 , piece_id=Q5X72MXPD2VX3TRLK37VB3OV5Y4WLD4P6CWPAVL2X4D6BSIIOFSQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802186428 (11h31m47.391531415s) [noise=true , address=104.36.82.5:4846 , piece_id=6NNDPTLCBYLGFWU2H57AIRLEEXTREPMET5R2YZETWQYPIP2ZUHHQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802550154 (11h31m47.391895141s) [noise=true , address=104.36.82.46:4501 , piece_id=CVPKLEXG5FMF3NAR3ONF7LGIWUXJ7OSF5BBOTJ3BLQE4CFFNQKBA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802702837 (11h31m47.392047814s) [noise=true , address=45.91.116.9:4268 , piece_id=VJP2ZZ3UMB2XMXGDGIV45YOLKHRNNJORX2NIB3SHBCI4UZNJEN7A, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802844608 (11h31m47.392189595s) [noise=true , address=104.36.82.16:4328 , piece_id=BFL3YNCSEKYNXJV6PJDPERHGUC6PDQZ52JNSLZDZPH3DCLYCAJPA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803081390 (11h31m47.392426377s) [noise=true , address=104.36.82.47:4523 , piece_id=4WH6OVD7YSUMLPUBHAQPDR5VYX7Y36WKS3MSM7ULY6CI73BALULQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803116328 (11h31m47.392461315s) [noise=true , address=45.91.116.11:4150 , piece_id=64KLQELYVF4SAJJYCFSQD3X5USFFYLXGNOJOQGZ36LOYOFEJ2UBQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803402735 (11h31m47.392747722s) [noise=true , address=104.36.82.12:4165 , piece_id=BBVGOIQK3IBDXSA5KXGHV4Z4TRPZYJQGJT3PYQJXZUI7ZJ55KRJA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803414909 (11h31m47.392759896s) [noise=true , address=104.36.82.39:4752 , piece_id=ITMK3WWPOHUBVDUV2ZUL65XLBGCIVZKGUD544R2K3FCAMTQN5TXQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803693041 (11h31m47.393038028s) [noise=true , address=104.36.82.36:4622 , piece_id=JYCNDCLZYJWGEB4ZWJM5ES3G25WPLBPOY6UGXI37SFC467KTIZEA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803929323 (11h31m47.39327431s) [noise=true , address=104.36.82.42:4813 , piece_id=UAXL4ZUQ4W6P3DLPKIWW7KC3U2MZPZQ25ZA7Z4A7BFQ42EMJZAXQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804163892 (11h31m47.393508879s) [noise=true , address=104.36.82.58:4269 , piece_id=QX6CM6DHSIW2KXY2Z6NRMSRG3HMBRNBQ3JRPHXBGOQMEL5HVZ5ZQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804383622 (11h31m47.393728609s) [noise=true , address=45.91.116.4:4969 , piece_id=GVQDU7BRCLA6PHFSRGOJZBAV6QZQRI2ZEG3EM6I2TJAZ4JAE4P7Q, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804621557 (11h31m47.393966544s) [noise=true , address=104.36.82.23:4453 , piece_id=BLRIGJ5Z3NNUPUHC2YZW4PDMK2SDUVPE75GO5T4WSFTT2FDOEEOQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804837270 (11h31m47.394182257s) [noise=true , address=104.36.82.2:4744 , piece_id=V6TRPHHDLGWKVLQH7BT6AVYVOKVVP3EHL67D62SPHAKNMWICYJWQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804911372 (11h31m47.394256359s) [noise=true , address=104.36.82.43:4849 , piece_id=IZH45DQEPVT7WEQ2UDAAA7OS37XKMKG4VBJRRUEKQ3G36YUM2LMQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.805071538 (11h31m47.394416515s) [noise=true , address=64.49.13.170:4728 , piece_id=LED5VGYENQQBGOP3UH4NKT6O6TH4M6OEZCIXZ5N7AYWFLW452KSA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.805261793 (11h31m47.39460677s) [noise=true , address=104.36.82.44:4899 , piece_id=I5PVQ6WHWORI2TFF2SJSZDAUDH3FX5DVNPP23VC6JECMDAO6ZM6Q, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:48.969580099 (11h31m48.558925086s) [upload=12022]: Next returned nil segment Aug 7 21:11:48.969596991 (11h31m48.558941978s) [upload=12022]: Waiting for error group managing segments... Aug 7 21:11:49.265222255 (11h31m48.854567242s) [noise=true , address=64.49.13.170:4728 , piece_id=LED5VGYENQQBGOP3UH4NKT6O6TH4M6OEZCIXZ5N7AYWFLW452KSA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.737280759 (11h31m49.326625746s) [noise=true , address=104.36.82.58:4269 , piece_id=QX6CM6DHSIW2KXY2Z6NRMSRG3HMBRNBQ3JRPHXBGOQMEL5HVZ5ZQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.753249862 (11h31m49.342594849s) [noise=true , address=104.36.82.12:4165 , piece_id=BBVGOIQK3IBDXSA5KXGHV4Z4TRPZYJQGJT3PYQJXZUI7ZJ55KRJA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.780910703 (11h31m49.37025569s) [noise=true , address=45.91.116.9:4268 , piece_id=VJP2ZZ3UMB2XMXGDGIV45YOLKHRNNJORX2NIB3SHBCI4UZNJEN7A, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.813098015 (11h31m49.402442992s) [noise=true , address=104.36.82.46:4501 , piece_id=CVPKLEXG5FMF3NAR3ONF7LGIWUXJ7OSF5BBOTJ3BLQE4CFFNQKBA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.827616893 (11h31m49.41696188s) [noise=true , address=104.36.82.42:4813 , piece_id=UAXL4ZUQ4W6P3DLPKIWW7KC3U2MZPZQ25ZA7Z4A7BFQ42EMJZAXQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.844644011 (11h31m49.433988998s) [noise=true , address=104.36.82.5:4846 , piece_id=6NNDPTLCBYLGFWU2H57AIRLEEXTREPMET5R2YZETWQYPIP2ZUHHQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.896979020 (11h31m49.486324007s) [noise=true , address=104.36.82.16:4328 , piece_id=BFL3YNCSEKYNXJV6PJDPERHGUC6PDQZ52JNSLZDZPH3DCLYCAJPA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.913302922 (11h31m49.502647909s) [noise=true , address=104.36.82.36:4622 , piece_id=JYCNDCLZYJWGEB4ZWJM5ES3G25WPLBPOY6UGXI37SFC467KTIZEA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.928574831 (11h31m49.517919809s) [noise=true , address=104.36.82.22:4541 , piece_id=Q5X72MXPD2VX3TRLK37VB3OV5Y4WLD4P6CWPAVL2X4D6BSIIOFSQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.967608171 (11h31m49.556953148s) [noise=true , address=104.36.82.43:4849 , piece_id=IZH45DQEPVT7WEQ2UDAAA7OS37XKMKG4VBJRRUEKQ3G36YUM2LMQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.087000568 (11h31m49.676345555s) [noise=true , address=45.91.116.11:4150 , piece_id=64KLQELYVF4SAJJYCFSQD3X5USFFYLXGNOJOQGZ36LOYOFEJ2UBQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.098037264 (11h31m49.687382241s) [noise=true , address=45.91.116.4:4969 , piece_id=GVQDU7BRCLA6PHFSRGOJZBAV6QZQRI2ZEG3EM6I2TJAZ4JAE4P7Q, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.160148408 (11h31m49.749493395s) [noise=true , address=104.36.82.47:4523 , piece_id=4WH6OVD7YSUMLPUBHAQPDR5VYX7Y36WKS3MSM7ULY6CI73BALULQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.160172064 (11h31m49.749517051s) [seg_pos={0 0}, upload=12022]: Segment reached optimal threshold of 14 pieces. Aug 7 21:11:50.160367668 (11h31m49.749712655s) [noise=true , address=104.36.82.44:4899 , piece_id=I5PVQ6WHWORI2TFF2SJSZDAUDH3FX5DVNPP23VC6JECMDAO6ZM6Q, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.160428794 (11h31m49.749773782s) [noise=true , address=104.36.82.23:4453 , piece_id=BLRIGJ5Z3NNUPUHC2YZW4PDMK2SDUVPE75GO5T4WSFTT2FDOEEOQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.160601615 (11h31m49.749946592s) [noise=true , address=104.36.82.2:4744 , piece_id=V6TRPHHDLGWKVLQH7BT6AVYVOKVVP3EHL67D62SPHAKNMWICYJWQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: read tcp 188.166.225.170:55148->104.36.82.2:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:11:50.160658343 (11h31m49.75000333s) [noise=true , address=104.36.82.39:4752 , piece_id=ITMK3WWPOHUBVDUV2ZUL65XLBGCIVZKGUD544R2K3FCAMTQN5TXQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: read tcp 188.166.225.170:46618->104.36.82.39:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:11:50.160688220 (11h31m49.750033207s) [seg_pos={0 0}, upload=12022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:11:50.160746132 (11h31m49.750091179s) [upload=12022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:11:50.160752644 (11h31m49.750097651s) [upload=12022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:50.684435488 (11h31m50.273780475s) [upload=12022]: Done uploading segments... Aug 7 21:11:50.684472599 (11h31m50.273817576s) [upload=12022]: Upload finished. err: <nil> Duration: 6.770556617s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.001053594s FAIL (2025-08-07 21:16:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7s) [noise=true , address=45.91.116.7:4034 , piece_id=RDFD2PLXXJ3GRAOR5ZLPKDLRK5IGJTZSBSNVG4A6XFZB75WPJ22Q, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234254518 (11h32m51.823599506s) [noise=true , address=104.36.82.44:4882 , piece_id=M7WROH24EKE2EMGFKDDDS6UOM3E5P4OXLT2QDIBXN2BCYFZCYUMA, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234402521 (11h32m51.823747508s) [noise=true , address=104.36.82.62:4427 , piece_id=QTWU24TIM66G6SQXGSU4H22JZTWDPYSZVIZV4FGRILO77SGECVNQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234520336 (11h32m51.823865313s) [noise=true , address=104.36.82.60:4348 , piece_id=KYGK6JIIPPPB3WXEJME7RHT4YU2K5YA6XO2UFS7RVRFC2MK5HL3Q, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234683298 (11h32m51.824028285s) [noise=true , address=104.36.82.41:4772 , piece_id=BGC5NOQVRQ4BQEUPDI5UDRVKUBSTZMOQKGKLTHLTBNWFTORUWZQQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234718776 (11h32m51.824063753s) [noise=true , address=104.36.82.34:4571 , piece_id=APKT66JAXIQNA5UXXM3FCX7JCN5ARKXPDUD6ZJ6WB46MMR2VF3UQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234915192 (11h32m51.824260179s) [noise=true , address=45.91.116.5:4325 , piece_id=P5KGQGDJXUKBS7F7ZK6TIP52U7I2OI74XFJTQSSQRFA2AS4WQDVQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.235686748 (11h32m51.825031735s) [noise=true , address=104.36.82.59:4333 , piece_id=BR6IGKYFNUAH2ZWKQMUQHYGMOL27HFFGN6PEPJOP7WODCYOVUJEQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:53.755589114 (11h32m53.344934101s) [upload=12039]: Next returned nil segment Aug 7 21:12:53.755603641 (11h32m53.344948629s) [upload=12039]: Waiting for error group managing segments... Aug 7 21:12:54.242895872 (11h32m53.832240859s) [noise=true , address=104.36.82.44:4882 , piece_id=M7WROH24EKE2EMGFKDDDS6UOM3E5P4OXLT2QDIBXN2BCYFZCYUMA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.247833398 (11h32m53.837178375s) [noise=true , address=104.36.82.23:4453 , piece_id=OIB2N3NXKLOBJS2UV353TMJMCMK3QM4UGLNKX7P6PUZDT2MRNILQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.248473462 (11h32m53.837818449s) [noise=true , address=104.36.82.16:4300 , piece_id=RVRXMZLRANZJMWSQAF7HTJUULYADEA2I6GPHDTFGKNS6FNWE36VA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.248750062 (11h32m53.83809505s) [noise=true , address=104.36.82.60:4348 , piece_id=KYGK6JIIPPPB3WXEJME7RHT4YU2K5YA6XO2UFS7RVRFC2MK5HL3Q, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.249467144 (11h32m53.838812131s) [noise=true , address=104.36.82.18:4378 , piece_id=ZJJ542SCBSUKTG7QB5PQFTDRPHMFHN32RY6LXNSDLU7X6SCEVBNQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.269168297 (11h32m53.858513304s) [noise=true , address=104.36.82.56:4247 , piece_id=X7EBBRNI7ZUNCR6SYWNS3TTBBQ4Z3BZZYWZNBYFBYRWYTV4TU5VQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.286754543 (11h32m53.87609953s) [noise=true , address=104.36.82.34:4571 , piece_id=APKT66JAXIQNA5UXXM3FCX7JCN5ARKXPDUD6ZJ6WB46MMR2VF3UQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.315127418 (11h32m53.904472395s) [noise=true , address=104.36.82.43:4866 , piece_id=AO5FAMAICZ2WFNV3L75ULE6UDCKGENWEBO47HXEA5ZVZ6KPOBFXQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.339819320 (11h32m53.929164307s) [noise=true , address=104.36.82.55:4195 , piece_id=CKLDMZFNPKVSRXP5A6Z7DIRQOTMLENLOL7XBDWJTMIRDNCFV3ZIA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.352457851 (11h32m53.941802858s) [noise=true , address=104.36.82.59:4333 , piece_id=BR6IGKYFNUAH2ZWKQMUQHYGMOL27HFFGN6PEPJOP7WODCYOVUJEQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.366096104 (11h32m53.955441091s) [noise=true , address=104.36.82.6:4890 , piece_id=DBUDIACR7PEFAFGSOBGEIMWWYDQ4QD4ZWFQ5MNIT2DRDSHWJUE7A, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.366979544 (11h32m53.956324531s) [noise=true , address=104.36.82.62:4427 , piece_id=QTWU24TIM66G6SQXGSU4H22JZTWDPYSZVIZV4FGRILO77SGECVNQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.381141350 (11h32m53.970486347s) [noise=true , address=104.36.82.41:4772 , piece_id=BGC5NOQVRQ4BQEUPDI5UDRVKUBSTZMOQKGKLTHLTBNWFTORUWZQQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.391573819 (11h32m53.980918806s) [noise=true , address=45.91.116.8:4067 , piece_id=GYAFL34K7CTRS3SOTQZFERYUQGQKVOBWUTPJ3E5CFYBZ7ON5RQVA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.391594779 (11h32m53.980939766s) [seg_pos={0 0}, upload=12039]: Segment reached optimal threshold of 14 pieces. Aug 7 21:12:54.391723245 (11h32m53.981068232s) [noise=true , address=45.91.116.7:4034 , piece_id=RDFD2PLXXJ3GRAOR5ZLPKDLRK5IGJTZSBSNVG4A6XFZB75WPJ22Q, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391809851 (11h32m53.981154838s) [noise=true , address=45.91.116.5:4325 , piece_id=P5KGQGDJXUKBS7F7ZK6TIP52U7I2OI74XFJTQSSQRFA2AS4WQDVQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391822284 (11h32m53.981167271s) [noise=true , address=104.36.82.45:4939 , piece_id=UIVN6HS5ZIB44VIN3PBQSMVEEFAYA3DF43KAALY6G6HCFADLZWIQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection; write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection Aug 7 21:12:54.391862171 (11h32m53.981207158s) [noise=true , address=104.36.82.43:4867 , piece_id=ZZCKCBXHV7POWCGCQ7DAJ2QE5A4UYFEERTSCUGVWU7UC76BYZTYA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zENB6bYmb9hqtGTcJ1afKVbcNtbEsGo4dpsSEdh2feqHifmHQ): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391878141 (11h32m53.981223118s) [seg_pos={0 0}, upload=12039]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:12:54.391977522 (11h32m53.981322509s) [upload=12039]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:12:54.391981659 (11h32m53.981326646s) [upload=12039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.855789778 (11h32m54.445134765s) [upload=12039]: Done uploading segments... Aug 7 21:12:54.855808144 (11h32m54.445153121s) [upload=12039]: Upload finished. err: <nil> Aug 7 21:13:45.303666216 (11h33m44.893011253s) [upload=12059]: Starting upload Aug 7 21:13:45.303786125 (11h33m44.893131112s) [upload=12059]: Done starting upload Aug 7 21:13:45.303856099 (11h33m44.893201126s) [upload=12059]: Uploading segments... Aug 7 21:13:45.304260994 (11h33m44.893605981s) [upload=12059]: Got next segment. Inline: false Aug 7 21:13:45.304300049 (11h33m44.893645036s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.304305609 (11h33m44.893650596s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:44.664782364 (11h34m44.254127351s) [upload=12059]: Next segment err: context deadline exceeded Aug 7 21:14:44.664969041 (11h34m44.254314028s) [upload=12059]: Done uploading segments... Aug 7 21:14:44.665004269 (11h34m44.254349256s) [upload=12059]: Upload finished. err: context deadline exceeded Aug 7 21:14:45.557957097 (11h34m45.147302084s) [upload=12072]: Starting upload Aug 7 21:14:45.558041418 (11h34m45.147386395s) [upload=12072]: Done starting upload Aug 7 21:14:45.558153183 (11h34m45.14749817s) [upload=12072]: Uploading segments... Aug 7 21:14:45.558236202 (11h34m45.147581189s) [upload=12072]: Got next segment. Inline: false Aug 7 21:14:45.558263704 (11h34m45.147608691s) [upload=12072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.558268123 (11h34m45.1476131s) [upload=12072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:44.909553125 (11h35m44.498898112s) [upload=12072]: Next segment err: context deadline exceeded Aug 7 21:15:44.909693113 (11h35m44.49903809s) [upload=12072]: Done uploading segments... Aug 7 21:15:44.909705847 (11h35m44.499050834s) [upload=12072]: Upload finished. err: context deadline exceeded Aug 7 21:15:45.780163446 (11h35m45.369508433s) [upload=12088]: Starting upload Aug 7 21:15:45.780230324 (11h35m45.369575311s) [upload=12088]: Done starting upload Aug 7 21:15:45.780343460 (11h35m45.369688497s) [upload=12088]: Uploading segments... Aug 7 21:15:45.780437410 (11h35m45.369782397s) [upload=12088]: Got next segment. Inline: false Aug 7 21:15:45.780481765 (11h35m45.369826772s) [upload=12088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:45.780488508 (11h35m45.369833535s) [upload=12088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:45.135036416 (11h36m44.724381403s) [upload=12088]: Next segment err: context deadline exceeded Aug 7 21:16:45.135160053 (11h36m44.72450504s) [upload=12088]: Done uploading segments... Aug 7 21:16:45.135188378 (11h36m44.724533375s) [upload=12088]: Upload finished. err: context deadline exceeded Duration: 1m0.001066028s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................X.XXX] duration: 1m0.001260139s FAIL (2025-08-07 21:16:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .851762080 (11h28m35.441107057s) [upload=11967]: Starting upload Aug 7 21:08:35.851828216 (11h28m35.441173203s) [upload=11967]: Done starting upload Aug 7 21:08:35.851888952 (11h28m35.441233939s) [upload=11967]: Uploading segments... Aug 7 21:08:35.851977722 (11h28m35.441322709s) [upload=11967]: Got next segment. Inline: true Aug 7 21:08:35.851983424 (11h28m35.441328411s) [upload=11967]: Waiting for error group managing segments... Aug 7 21:08:35.852017519 (11h28m35.441362506s) [upload=11967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:08:35.852021105 (11h28m35.441366093s) [upload=11967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:08:35.852023460 (11h28m35.441368447s) [upload=11967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:08:36.115286628 (11h28m35.704631615s) [upload=11967]: Done uploading segments... Aug 7 21:08:36.115331112 (11h28m35.704676099s) [upload=11967]: Upload finished. err: <nil> Aug 7 21:09:20.788493181 (11h29m20.377838178s) [upload=11976]: Starting upload Aug 7 21:09:20.788595918 (11h29m20.377940965s) [upload=11976]: Done starting upload Aug 7 21:09:20.788684948 (11h29m20.378029975s) [upload=11976]: Uploading segments... Aug 7 21:09:20.788747948 (11h29m20.378092935s) [upload=11976]: Got next segment. Inline: true Aug 7 21:09:20.788756344 (11h29m20.378101361s) [upload=11976]: Waiting for error group managing segments... Aug 7 21:09:20.788807752 (11h29m20.378152739s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:20.788813985 (11h29m20.378158972s) [upload=11976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:20.788818584 (11h29m20.378163611s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:21.070630248 (11h29m20.659975235s) [upload=11976]: Done uploading segments... Aug 7 21:09:21.070652190 (11h29m20.659997247s) [upload=11976]: Upload finished. err: <nil> Aug 7 21:09:46.948530536 (11h29m46.537875523s) [upload=11987]: Starting upload Aug 7 21:09:46.948588607 (11h29m46.537933584s) [upload=11987]: Done starting upload Aug 7 21:09:46.948642600 (11h29m46.537987587s) [upload=11987]: Uploading segments... Aug 7 21:09:46.948702786 (11h29m46.538047773s) [upload=11987]: Got next segment. Inline: true Aug 7 21:09:46.948707776 (11h29m46.538052763s) [upload=11987]: Waiting for error group managing segments... Aug 7 21:09:46.948747241 (11h29m46.538092228s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:46.948752340 (11h29m46.538097327s) [upload=11987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:46.948754896 (11h29m46.538099883s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.214261274 (11h29m46.803606261s) [upload=11987]: Done uploading segments... Aug 7 21:09:47.214279799 (11h29m46.803624786s) [upload=11987]: Upload finished. err: <nil> Aug 7 21:10:37.572355395 (11h30m37.161700372s) [upload=11999]: Starting upload Aug 7 21:10:37.572410881 (11h30m37.161755868s) [upload=11999]: Done starting upload Aug 7 21:10:37.572467810 (11h30m37.161812797s) [upload=11999]: Uploading segments... Aug 7 21:10:37.572531982 (11h30m37.161876959s) [upload=11999]: Got next segment. Inline: true Aug 7 21:10:37.572536320 (11h30m37.161881307s) [upload=11999]: Waiting for error group managing segments... Aug 7 21:10:37.572570917 (11h30m37.161915904s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:37.572574955 (11h30m37.161919942s) [upload=11999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:37.572577520 (11h30m37.161922507s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:37.838997375 (11h30m37.428342362s) [upload=11999]: Done uploading segments... Aug 7 21:10:37.839014699 (11h30m37.428359686s) [upload=11999]: Upload finished. err: <nil> Aug 7 21:11:31.258043368 (11h31m30.847388355s) [upload=12015]: Starting upload Aug 7 21:11:31.258101790 (11h31m30.847446777s) [upload=12015]: Done starting upload Aug 7 21:11:31.258163838 (11h31m30.847508825s) [upload=12015]: Uploading segments... Aug 7 21:11:31.258195730 (11h31m30.847540717s) [upload=12015]: Got next segment. Inline: true Aug 7 21:11:31.258199567 (11h31m30.847544565s) [upload=12015]: Waiting for error group managing segments... Aug 7 21:11:31.258240175 (11h31m30.847585162s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.258244853 (11h31m30.847589841s) [upload=12015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.258247358 (11h31m30.847592345s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.535852595 (11h31m31.125197582s) [upload=12015]: Done uploading segments... Aug 7 21:11:31.535871041 (11h31m31.125216028s) [upload=12015]: Upload finished. err: <nil> Aug 7 21:11:57.912240700 (11h31m57.501585687s) [upload=12025]: Starting upload Aug 7 21:11:57.912288532 (11h31m57.501633519s) [upload=12025]: Done starting upload Aug 7 21:11:57.912345871 (11h31m57.501690858s) [upload=12025]: Uploading segments... Aug 7 21:11:57.912374877 (11h31m57.501719874s) [upload=12025]: Got next segment. Inline: true Aug 7 21:11:57.912378794 (11h31m57.501723781s) [upload=12025]: Waiting for error group managing segments... Aug 7 21:11:57.912425353 (11h31m57.501770401s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:57.912431234 (11h31m57.501776232s) [upload=12025]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:57.912435933 (11h31m57.50178095s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:57.261749711 (11h32m56.851094698s) [upload=12025]: Done uploading segments... Aug 7 21:12:57.261780350 (11h32m56.851125337s) [upload=12025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:07.178414027 (11h33m6.767759014s) [upload=12044]: Starting upload Aug 7 21:13:07.178467459 (11h33m6.767812446s) [upload=12044]: Done starting upload Aug 7 21:13:07.178514029 (11h33m6.767859016s) [upload=12044]: Uploading segments... Aug 7 21:13:07.178543715 (11h33m6.767888703s) [upload=12044]: Got next segment. Inline: true Aug 7 21:13:07.178547282 (11h33m6.767892259s) [upload=12044]: Waiting for error group managing segments... Aug 7 21:13:07.178580015 (11h33m6.767925002s) [upload=12044]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:07.178583561 (11h33m6.767928548s) [upload=12044]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:07.178585735 (11h33m6.767930722s) [upload=12044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:07.471379642 (11h33m7.060724629s) [upload=12044]: Done uploading segments... Aug 7 21:13:07.471396245 (11h33m7.060741232s) [upload=12044]: Upload finished. err: <nil> Aug 7 21:13:32.020845488 (11h33m31.610190465s) [upload=12052]: Starting upload Aug 7 21:13:32.020909792 (11h33m31.610254769s) [upload=12052]: Done starting upload Aug 7 21:13:32.020999433 (11h33m31.61034443s) [upload=12052]: Uploading segments... Aug 7 21:13:32.021133279 (11h33m31.610478267s) [upload=12052]: Got next segment. Inline: true Aug 7 21:13:32.021141214 (11h33m31.610486231s) [upload=12052]: Waiting for error group managing segments... Aug 7 21:13:32.021222079 (11h33m31.610567066s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:32.021229403 (11h33m31.61057439s) [upload=12052]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:32.021234362 (11h33m31.610579399s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:31.378507697 (11h34m30.967852684s) [upload=12052]: Done uploading segments... Aug 7 21:14:31.378529929 (11h34m30.967874916s) [upload=12052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:36.610398147 (11h34m36.199743124s) [upload=12069]: Starting upload Aug 7 21:14:36.610597218 (11h34m36.199942205s) [upload=12069]: Done starting upload Aug 7 21:14:36.610671039 (11h34m36.200016016s) [upload=12069]: Uploading segments... Aug 7 21:14:36.610800146 (11h34m36.200145133s) [upload=12069]: Got next segment. Inline: true Aug 7 21:14:36.610805957 (11h34m36.200150944s) [upload=12069]: Waiting for error group managing segments... Aug 7 21:14:36.610869649 (11h34m36.200214636s) [upload=12069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:36.610874759 (11h34m36.200219746s) [upload=12069]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:36.610877584 (11h34m36.200222571s) [upload=12069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:35.947737631 (11h35m35.537082618s) [upload=12069]: Done uploading segments... Aug 7 21:15:35.947756376 (11h35m35.537101353s) [upload=12069]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:45.995643042 (11h35m45.584988019s) [upload=12089]: Starting upload Aug 7 21:15:45.995767078 (11h35m45.585112086s) [upload=12089]: Done starting upload Aug 7 21:15:45.995846110 (11h35m45.585191127s) [upload=12089]: Uploading segments... Aug 7 21:15:45.995939870 (11h35m45.585284917s) [upload=12089]: Got next segment. Inline: true Aug 7 21:15:45.995949598 (11h35m45.585294615s) [upload=12089]: Waiting for error group managing segments... Aug 7 21:15:45.996009222 (11h35m45.585354249s) [upload=12089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:45.996016616 (11h35m45.585361623s) [upload=12089]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:45.996022197 (11h35m45.585367204s) [upload=12089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:45.347388198 (11h36m44.936733185s) [upload=12089]: Done uploading segments... Aug 7 21:16:45.347412374 (11h36m44.936757362s) [upload=12089]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001285276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................X..XXXX] duration: 30.001054387s FAIL (2025-08-07 21:16:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573225655 (11h33m16.162570642s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573527402 (11h33m16.162872389s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573918310 (11h33m16.163263297s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574198737 (11h33m16.163543724s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574518488 (11h33m16.163863465s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574831648 (11h33m16.164176635s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:18.308274980 (11h33m17.897619967s) [upload=12046]: Next returned nil segment Aug 7 21:13:18.308295720 (11h33m17.897640707s) [upload=12046]: Waiting for error group managing segments... Aug 7 21:13:18.656166252 (11h33m18.245511229s) [noise=true , address=77.90.31.13:4626 , piece_id=S4OKGN6SKZ7GLYKCT3S4QSQFK75FX3KAXGQAOORJRVDJ7HV5UTWQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.668187531 (11h33m18.257532518s) [noise=true , address=104.36.82.6:4873 , piece_id=VLAKUO36OHVGPY3Q723O5OUKURJ2SWREZC3SABUX2J7O6MBM7Y2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.721769396 (11h33m18.311114384s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.741305974 (11h33m18.330650951s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.780529499 (11h33m18.369874486s) [noise=true , address=77.90.31.11:4553 , piece_id=4QVTWAAY3JK3EQJIHK7ANW6E4UYUI3KXCHZOWRK75JHZZRPZ6ENQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.865633524 (11h33m18.454978511s) [noise=true , address=45.91.116.11:4119 , piece_id=6PDH442NIXRNQCIFX2Y4PNKU5C73T2H6KNHKGMIW2UGCTC5NX7ZA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880648330 (11h33m18.469993317s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880718855 (11h33m18.470063842s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.935042530 (11h33m18.524387517s) [noise=true , address=45.91.116.5:4298 , piece_id=Y4FCHPKGJXDGWLH2FT6M5PJV5RA4UOYFQ3SHTWFJN4WO3FTFFG2A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.969979703 (11h33m18.55932468s) [noise=true , address=45.91.116.5:4299 , piece_id=5YQ5OJAPBA3N2KMYSVWQCNOTH7QPX2WCHHUXTLNSVNH2P7MZEJ3A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.972928735 (11h33m18.562273722s) [noise=true , address=45.91.116.12:4226 , piece_id=5QFSWOPQLZO2KEP4J27RDULUD4VCQSIE6BZ6BO3MFVFAP67OMW7A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.047677267 (11h33m18.637022254s) [noise=true , address=104.36.82.15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082384121 (11h33m18.671729108s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082409660 (11h33m18.671754647s) [seg_pos={0 0}, upload=12046]: Segment reached optimal threshold of 13 pieces. Aug 7 21:13:19.082628518 (11h33m18.671973495s) [noise=true , address=77.90.31.14:4661 , piece_id=P4DQFG2ACQKL52JQZFCTIFCUHFPLDIYEOOG4OPBJY4MIXBVKCYDQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Aug 7 21:13:19.082795017 (11h33m18.672139994s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: read tcp 188.166.225.170:56758->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082822870 (11h33m18.672167857s) [noise=true , address=104.36.82.16:4294 , piece_id=QN2G4ZPHQ2VBVMEJFPY3OLOGLHZRKEFC4SXOXSHFBQMD47QMAM7Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:47752->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082839501 (11h33m18.672184478s) [seg_pos={0 0}, upload=12046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:13:19.082910688 (11h33m18.672255675s) [upload=12046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:19.082914756 (11h33m18.672259743s) [upload=12046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:19.387715025 (11h33m18.977060012s) [upload=12046]: Done uploading segments... Aug 7 21:13:19.387739612 (11h33m18.977084599s) [upload=12046]: Upload finished. err: <nil> Aug 7 21:13:45.069592425 (11h33m44.658937412s) [upload=12058]: Starting upload Aug 7 21:13:45.069646459 (11h33m44.658991446s) [upload=12058]: Done starting upload Aug 7 21:13:45.069729608 (11h33m44.659074595s) [upload=12058]: Uploading segments... Aug 7 21:13:45.069894263 (11h33m44.65923925s) [upload=12058]: Got next segment. Inline: false Aug 7 21:13:45.069927166 (11h33m44.659272153s) [upload=12058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.069931384 (11h33m44.659276371s) [upload=12058]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:14.428211569 (11h34m14.017556556s) [upload=12058]: Next segment err: context deadline exceeded Aug 7 21:14:14.428287775 (11h34m14.017632762s) [upload=12058]: Done uploading segments... Aug 7 21:14:14.428338853 (11h34m14.01768384s) [upload=12058]: Upload finished. err: context deadline exceeded Aug 7 21:14:49.077304183 (11h34m48.66664917s) [upload=12074]: Starting upload Aug 7 21:14:49.077411358 (11h34m48.666756365s) [upload=12074]: Done starting upload Aug 7 21:14:49.077536547 (11h34m48.666881534s) [upload=12074]: Uploading segments... Aug 7 21:14:49.077698226 (11h34m48.667043214s) [upload=12074]: Got next segment. Inline: false Aug 7 21:14:49.077743253 (11h34m48.66708824s) [upload=12074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:49.077748543 (11h34m48.66709353s) [upload=12074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:18.426190970 (11h35m18.015535997s) [upload=12074]: Next segment err: context deadline exceeded Aug 7 21:15:18.426350985 (11h35m18.015695972s) [upload=12074]: Done uploading segments... Aug 7 21:15:18.426379730 (11h35m18.015724707s) [upload=12074]: Upload finished. err: context deadline exceeded Aug 7 21:15:36.581505585 (11h35m36.170850572s) [upload=12086]: Starting upload Aug 7 21:15:36.581560250 (11h35m36.170905227s) [upload=12086]: Done starting upload Aug 7 21:15:36.581682133 (11h35m36.17102711s) [upload=12086]: Uploading segments... Aug 7 21:15:36.581769470 (11h35m36.171114457s) [upload=12086]: Got next segment. Inline: false Aug 7 21:15:36.581824996 (11h35m36.171169983s) [upload=12086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:36.581829786 (11h35m36.171174773s) [upload=12086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:05.932290909 (11h36m5.521635897s) [upload=12086]: Next segment err: context deadline exceeded Aug 7 21:16:05.932368198 (11h36m5.521713185s) [upload=12086]: Done uploading segments... Aug 7 21:16:05.932396572 (11h36m5.521741559s) [upload=12086]: Upload finished. err: context deadline exceeded Aug 7 21:16:06.820264050 (11h36m6.409609037s) [upload=12095]: Starting upload Aug 7 21:16:06.820316621 (11h36m6.409661608s) [upload=12095]: Done starting upload Aug 7 21:16:06.820429747 (11h36m6.409774734s) [upload=12095]: Uploading segments... Aug 7 21:16:06.820494201 (11h36m6.409839188s) [upload=12095]: Got next segment. Inline: false Aug 7 21:16:06.820519059 (11h36m6.409864046s) [upload=12095]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:06.820524339 (11h36m6.409869326s) [upload=12095]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:36.174935298 (11h36m35.764280285s) [upload=12095]: Next segment err: context deadline exceeded Aug 7 21:16:36.175145370 (11h36m35.764490357s) [upload=12095]: Done uploading segments... Aug 7 21:16:36.175169056 (11h36m35.764514043s) [upload=12095]: Upload finished. err: context deadline exceeded Duration: 30.001064085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.001233491s FAIL (2025-08-07 21:16:34 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001245564s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....XX.......................X.............................] duration: 1.848280106s FAIL (2025-08-07 21:16:27 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== pb.BatchRequestItem_ObjectCommit Aug 7 20:43:37.067799677 (11h3m36.657144664s) [upload=11534]: Done uploading segments... Aug 7 20:43:37.067817170 (11h3m36.657162157s) [upload=11534]: Upload finished. err: <nil> Aug 7 20:44:07.448276030 (11h4m7.037621017s) [upload=11539]: Starting upload Aug 7 20:44:07.448326927 (11h4m7.037671904s) [upload=11539]: Done starting upload Aug 7 20:44:07.448378606 (11h4m7.037723593s) [upload=11539]: Uploading segments... Aug 7 20:44:07.448408994 (11h4m7.037753981s) [upload=11539]: Got next segment. Inline: true Aug 7 20:44:07.448414504 (11h4m7.037759491s) [upload=11539]: Waiting for error group managing segments... Aug 7 20:44:07.448448068 (11h4m7.037793055s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:44:07.448451785 (11h4m7.037796762s) [upload=11539]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:44:07.448454070 (11h4m7.037799057s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:44:07.799942249 (11h4m7.389287236s) [upload=11539]: Done uploading segments... Aug 7 20:44:07.799959912 (11h4m7.389304899s) [upload=11539]: Upload finished. err: <nil> Aug 7 20:45:19.440040944 (11h5m19.029385921s) [upload=11564]: Starting upload Aug 7 20:45:19.440090409 (11h5m19.029435386s) [upload=11564]: Done starting upload Aug 7 20:45:19.440146045 (11h5m19.029491032s) [upload=11564]: Uploading segments... Aug 7 20:45:19.440190359 (11h5m19.029535346s) [upload=11564]: Got next segment. Inline: true Aug 7 20:45:19.440194267 (11h5m19.029539254s) [upload=11564]: Waiting for error group managing segments... Aug 7 20:45:19.440226098 (11h5m19.029571085s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:45:19.440230026 (11h5m19.029575013s) [upload=11564]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:45:19.440232290 (11h5m19.029577277s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:45:19.858443364 (11h5m19.447788352s) [upload=11564]: Done uploading segments... Aug 7 20:45:19.858467080 (11h5m19.447812067s) [upload=11564]: Upload finished. err: <nil> Aug 7 20:46:00.840349205 (11h6m0.429694192s) [upload=11575]: Starting upload Aug 7 20:46:00.840387950 (11h6m0.429732927s) [upload=11575]: Done starting upload Aug 7 20:46:00.840431102 (11h6m0.429776079s) [upload=11575]: Uploading segments... Aug 7 20:46:00.840473594 (11h6m0.429818581s) [upload=11575]: Got next segment. Inline: true Aug 7 20:46:00.840477311 (11h6m0.429822298s) [upload=11575]: Waiting for error group managing segments... Aug 7 20:46:00.840510584 (11h6m0.429855571s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:00.840514551 (11h6m0.429859539s) [upload=11575]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:00.840516746 (11h6m0.429861733s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:01.149526394 (11h6m0.738871381s) [upload=11575]: Done uploading segments... Aug 7 20:46:01.149544077 (11h6m0.738889064s) [upload=11575]: Upload finished. err: <nil> Aug 7 20:46:32.531571609 (11h6m32.120916586s) [upload=11585]: Starting upload Aug 7 20:46:32.531627656 (11h6m32.120972643s) [upload=11585]: Done starting upload Aug 7 20:46:32.531679334 (11h6m32.121024311s) [upload=11585]: Uploading segments... Aug 7 20:46:32.531716416 (11h6m32.121061403s) [upload=11585]: Got next segment. Inline: true Aug 7 20:46:32.531720834 (11h6m32.121065821s) [upload=11585]: Waiting for error group managing segments... Aug 7 20:46:32.531758977 (11h6m32.121103964s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:32.531762984 (11h6m32.121107961s) [upload=11585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:32.531766161 (11h6m32.121111148s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:32.823368074 (11h6m32.412713051s) [upload=11585]: Done uploading segments... Aug 7 20:46:32.823385747 (11h6m32.412730724s) [upload=11585]: Upload finished. err: <nil> Aug 7 20:46:53.991475826 (11h6m53.580820813s) [upload=11590]: Starting upload Aug 7 20:46:53.991528908 (11h6m53.580873895s) [upload=11590]: Done starting upload Aug 7 20:46:53.991580286 (11h6m53.580925273s) [upload=11590]: Uploading segments... Aug 7 20:46:53.991609822 (11h6m53.58095481s) [upload=11590]: Got next segment. Inline: true Aug 7 20:46:53.991613780 (11h6m53.580958767s) [upload=11590]: Waiting for error group managing segments... Aug 7 20:46:53.991645681 (11h6m53.580990668s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:53.991649578 (11h6m53.580994565s) [upload=11590]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:53.991651822 (11h6m53.580996809s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:54.273031788 (11h6m53.862376835s) [upload=11590]: Done uploading segments... Aug 7 20:46:54.273056015 (11h6m53.862401032s) [upload=11590]: Upload finished. err: <nil> Aug 7 20:48:05.936480351 (11h8m5.525825338s) [upload=11610]: Starting upload Aug 7 20:48:05.936525999 (11h8m5.525870986s) [upload=11610]: Done starting upload Aug 7 20:48:05.936586324 (11h8m5.525931311s) [upload=11610]: Uploading segments... Aug 7 20:48:05.936615510 (11h8m5.525960487s) [upload=11610]: Got next segment. Inline: true Aug 7 20:48:05.936618666 (11h8m5.525963653s) [upload=11610]: Waiting for error group managing segments... Aug 7 20:48:05.936688099 (11h8m5.526033086s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:48:05.936692588 (11h8m5.526037575s) [upload=11610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:48:05.936694892 (11h8m5.526039879s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:48:06.223792648 (11h8m5.813137635s) [upload=11610]: Done uploading segments... Aug 7 20:48:06.223807756 (11h8m5.813152743s) [upload=11610]: Upload finished. err: <nil> Aug 7 20:49:00.370365369 (11h8m59.959710356s) [upload=11624]: Starting upload Aug 7 20:49:00.370414633 (11h8m59.95975962s) [upload=11624]: Done starting upload Aug 7 20:49:00.370467765 (11h8m59.959812752s) [upload=11624]: Uploading segments... Aug 7 20:49:00.370501249 (11h8m59.959846236s) [upload=11624]: Got next segment. Inline: true Aug 7 20:49:00.370504986 (11h8m59.959849973s) [upload=11624]: Waiting for error group managing segments... Aug 7 20:49:00.370538900 (11h8m59.959883887s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:00.370542898 (11h8m59.959887885s) [upload=11624]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:00.370545453 (11h8m59.95989043s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:00.736326238 (11h9m0.325671225s) [upload=11624]: Done uploading segments... Aug 7 20:49:00.736343821 (11h9m0.325688808s) [upload=11624]: Upload finished. err: <nil> Aug 7 20:49:43.904464651 (11h9m43.493809639s) [upload=11637]: Starting upload Aug 7 20:49:43.904540737 (11h9m43.493885724s) [upload=11637]: Done starting upload Aug 7 20:49:43.904599109 (11h9m43.493944096s) [upload=11637]: Uploading segments... Aug 7 20:49:43.904635608 (11h9m43.493980595s) [upload=11637]: Got next segment. Inline: true Aug 7 20:49:43.904639476 (11h9m43.493984463s) [upload=11637]: Waiting for error group managing segments... Aug 7 20:49:43.904675195 (11h9m43.494020182s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:43.904679252 (11h9m43.494024239s) [upload=11637]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:43.904683430 (11h9m43.494028417s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:44.269042214 (11h9m43.858387201s) [upload=11637]: Done uploading segments... Aug 7 20:49:44.269058595 (11h9m43.858403582s) [upload=11637]: Upload finished. err: <nil> Aug 7 20:50:51.186360319 (11h10m50.775705306s) [upload=11656]: Starting upload Aug 7 20:50:51.186464308 (11h10m50.775809295s) [upload=11656]: Done starting upload Aug 7 20:50:51.186527319 (11h10m50.775872306s) [upload=11656]: Uploading segments... Aug 7 20:50:51.186558759 (11h10m50.775903746s) [upload=11656]: Got next segment. Inline: true Aug 7 20:50:51.186562295 (11h10m50.775907282s) [upload=11656]: Waiting for error group managing segments... Aug 7 20:50:51.186595108 (11h10m50.775940095s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:50:51.186599105 (11h10m50.775944082s) [upload=11656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:50:51.186601591 (11h10m50.775946578s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:50:51.466357188 (11h10m51.055702175s) [upload=11656]: Done uploading segments... Aug 7 20:50:51.466375784 (11h10m51.055720781s) [upload=11656]: Upload finished. err: <nil> Aug 7 20:51:54.183767656 (11h11m53.773112643s) [upload=11674]: Starting upload Aug 7 20:51:54.183810869 (11h11m53.773155846s) [upload=11674]: Done starting upload Aug 7 20:51:54.183850504 (11h11m53.773195481s) [upload=11674]: Uploading segments... Aug 7 20:51:54.183877035 (11h11m53.773222012s) [upload=11674]: Got next segment. Inline: true Aug 7 20:51:54.183880662 (11h11m53.773225639s) [upload=11674]: Waiting for error group managing segments... Aug 7 20:51:54.183910529 (11h11m53.773255516s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:51:54.183914426 (11h11m53.773259413s) [upload=11674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:51:54.183916961 (11h11m53.773261938s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:51:54.662596438 (11h11m54.251941425s) [upload=11674]: Done uploading segments... Aug 7 20:51:54.662619702 (11h11m54.251964689s) [upload=11674]: Upload finished. err: <nil> Duration:* 1.8483092s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.00170705s FAIL (2025-08-07 21:16:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001721237s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.001181524s FAIL (2025-08-07 21:16:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 55442 (11h29m46.728800429s) [upload=11988]: Starting upload Aug 7 21:09:47.139503314 (11h29m46.728848301s) [upload=11988]: Done starting upload Aug 7 21:09:47.139548831 (11h29m46.728893818s) [upload=11988]: Uploading segments... Aug 7 21:09:47.139579450 (11h29m46.728924437s) [upload=11988]: Got next segment. Inline: true Aug 7 21:09:47.139583277 (11h29m46.728928264s) [upload=11988]: Waiting for error group managing segments... Aug 7 21:09:47.139618274 (11h29m46.728963261s) [upload=11988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:47.139622451 (11h29m46.728967439s) [upload=11988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:47.139624797 (11h29m46.728969784s) [upload=11988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.434297548 (11h29m47.023642535s) [upload=11988]: Done uploading segments... Aug 7 21:09:47.434326894 (11h29m47.023671872s) [upload=11988]: Upload finished. err: <nil> Aug 7 21:10:17.433462616 (11h30m17.022807593s) [upload=11995]: Starting upload Aug 7 21:10:17.433520777 (11h30m17.022865764s) [upload=11995]: Done starting upload Aug 7 21:10:17.433584859 (11h30m17.022929846s) [upload=11995]: Uploading segments... Aug 7 21:10:17.433692035 (11h30m17.023037012s) [upload=11995]: Got next segment. Inline: true Aug 7 21:10:17.433696885 (11h30m17.023041872s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:10:17.433744876 (11h30m17.023089853s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:17.433749755 (11h30m17.023094742s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:17.433754204 (11h30m17.023099191s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:17.710858662 (11h30m17.300203709s) [upload=11995]: Done uploading segments... Aug 7 21:10:17.710881997 (11h30m17.300227024s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:10:36.218517401 (11h30m35.807862378s) [upload=11998]: Starting upload Aug 7 21:10:36.218649954 (11h30m35.807994941s) [upload=11998]: Done starting upload Aug 7 21:10:36.218713896 (11h30m35.808058913s) [upload=11998]: Uploading segments... Aug 7 21:10:36.218766106 (11h30m35.808111103s) [upload=11998]: Got next segment. Inline: true Aug 7 21:10:36.218772398 (11h30m35.808117425s) [upload=11998]: Waiting for error group managing segments... Aug 7 21:10:36.218820571 (11h30m35.808165558s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:36.218825550 (11h30m35.808170537s) [upload=11998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:36.218828946 (11h30m35.808173943s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:36.494741534 (11h30m36.084086521s) [upload=11998]: Done uploading segments... Aug 7 21:10:36.494760030 (11h30m36.084105007s) [upload=11998]: Upload finished. err: <nil> Aug 7 21:11:36.918559943 (11h31m36.50790493s) [upload=12018]: Starting upload Aug 7 21:11:36.918615830 (11h31m36.507960817s) [upload=12018]: Done starting upload Aug 7 21:11:36.918660295 (11h31m36.508005282s) [upload=12018]: Uploading segments... Aug 7 21:11:36.918688629 (11h31m36.508033606s) [upload=12018]: Got next segment. Inline: true Aug 7 21:11:36.918691745 (11h31m36.508036732s) [upload=12018]: Waiting for error group managing segments... Aug 7 21:11:36.918721552 (11h31m36.508066539s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:36.918725530 (11h31m36.508070517s) [upload=12018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:36.918727794 (11h31m36.508072781s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:37.211063805 (11h31m36.800408792s) [upload=12018]: Done uploading segments... Aug 7 21:11:37.211087029 (11h31m36.800432026s) [upload=12018]: Upload finished. err: <nil> Aug 7 21:12:15.260577538 (11h32m14.849922525s) [upload=12032]: Starting upload Aug 7 21:12:15.260632023 (11h32m14.84997701s) [upload=12032]: Done starting upload Aug 7 21:12:15.260678571 (11h32m14.850023548s) [upload=12032]: Uploading segments... Aug 7 21:12:15.260706434 (11h32m14.850051421s) [upload=12032]: Got next segment. Inline: true Aug 7 21:12:15.260710011 (11h32m14.850054998s) [upload=12032]: Waiting for error group managing segments... Aug 7 21:12:15.260743897 (11h32m14.850088874s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:15.260747644 (11h32m14.850092631s) [upload=12032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:15.260750389 (11h32m14.850095376s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:15.565323071 (11h32m15.154668058s) [upload=12032]: Done uploading segments... Aug 7 21:12:15.565341296 (11h32m15.154686283s) [upload=12032]: Upload finished. err: <nil> Aug 7 21:12:41.001560374 (11h32m40.590905361s) [upload=12034]: Starting upload Aug 7 21:12:41.001659143 (11h32m40.59100413s) [upload=12034]: Done starting upload Aug 7 21:12:41.001745669 (11h32m40.591090666s) [upload=12034]: Uploading segments... Aug 7 21:12:41.001852704 (11h32m40.591197701s) [upload=12034]: Got next segment. Inline: true Aug 7 21:12:41.001857372 (11h32m40.591202359s) [upload=12034]: Waiting for error group managing segments... Aug 7 21:12:41.001907849 (11h32m40.591252836s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:41.001913169 (11h32m40.591258156s) [upload=12034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:41.001915584 (11h32m40.591260571s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:41.286244505 (11h32m40.875589493s) [upload=12034]: Done uploading segments... Aug 7 21:12:41.286260567 (11h32m40.875605544s) [upload=12034]: Upload finished. err: <nil> Aug 7 21:13:16.624584146 (11h33m16.213929123s) [upload=12047]: Starting upload Aug 7 21:13:16.624659630 (11h33m16.214004607s) [upload=12047]: Done starting upload Aug 7 21:13:16.624710127 (11h33m16.214055114s) [upload=12047]: Uploading segments... Aug 7 21:13:16.624745715 (11h33m16.214090702s) [upload=12047]: Got next segment. Inline: true Aug 7 21:13:16.624750654 (11h33m16.214095641s) [upload=12047]: Waiting for error group managing segments... Aug 7 21:13:16.624782977 (11h33m16.214127964s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:16.624786804 (11h33m16.214131791s) [upload=12047]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:16.624789339 (11h33m16.214134326s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.974391598 (11h33m45.563736596s) [upload=12047]: Done uploading segments... Aug 7 21:13:45.974419923 (11h33m45.56376491s) [upload=12047]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:31.328315035 (11h34m30.917660012s) [upload=12067]: Starting upload Aug 7 21:14:31.328377385 (11h34m30.917722373s) [upload=12067]: Done starting upload Aug 7 21:14:31.328450665 (11h34m30.917795652s) [upload=12067]: Uploading segments... Aug 7 21:14:31.328587126 (11h34m30.917932113s) [upload=12067]: Got next segment. Inline: true Aug 7 21:14:31.328592336 (11h34m30.917937323s) [upload=12067]: Waiting for error group managing segments... Aug 7 21:14:31.328654706 (11h34m30.917999693s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:31.328660026 (11h34m30.918005013s) [upload=12067]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:31.328662631 (11h34m30.918007618s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:00.687780769 (11h35m0.277125756s) [upload=12067]: Done uploading segments... Aug 7 21:15:00.687819252 (11h35m0.27716424s) [upload=12067]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:17.371028003 (11h35m16.96037298s) [upload=12080]: Starting upload Aug 7 21:15:17.371083610 (11h35m16.960428587s) [upload=12080]: Done starting upload Aug 7 21:15:17.371134738 (11h35m16.960479725s) [upload=12080]: Uploading segments... Aug 7 21:15:17.371166228 (11h35m16.960511215s) [upload=12080]: Got next segment. Inline: true Aug 7 21:15:17.371170035 (11h35m16.960515022s) [upload=12080]: Waiting for error group managing segments... Aug 7 21:15:17.371203299 (11h35m16.960548286s) [upload=12080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:17.371207227 (11h35m16.960552214s) [upload=12080]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:17.371209821 (11h35m16.960554809s) [upload=12080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:46.717162709 (11h35m46.306507686s) [upload=12080]: Done uploading segments... Aug 7 21:15:46.717181444 (11h35m46.306526431s) [upload=12080]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:47.625517071 (11h35m47.214862058s) [upload=12090]: Starting upload Aug 7 21:15:47.625581626 (11h35m47.214926613s) [upload=12090]: Done starting upload Aug 7 21:15:47.625639957 (11h35m47.214984944s) [upload=12090]: Uploading segments... Aug 7 21:15:47.625677468 (11h35m47.215022456s) [upload=12090]: Got next segment. Inline: true Aug 7 21:15:47.625681686 (11h35m47.215026673s) [upload=12090]: Waiting for error group managing segments... Aug 7 21:15:47.625720211 (11h35m47.215065198s) [upload=12090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:47.625724749 (11h35m47.215069736s) [upload=12090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:47.625727995 (11h35m47.215072982s) [upload=12090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:16.958194553 (11h36m16.547539541s) [upload=12090]: Done uploading segments... Aug 7 21:16:16.958214662 (11h36m16.547559659s) [upload=12090]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001192735s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.000866039s FAIL (2025-08-07 21:16:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000886037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XX...........X.......................X....] duration: 6.765227144s FAIL (2025-08-07 21:16:07 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 7106407 (11h31m46.936451394s) [upload=12022]: Uploading segments... Aug 7 21:11:47.347198184 (11h31m46.936543161s) [upload=12022]: Got next segment. Inline: false Aug 7 21:11:47.347234122 (11h31m46.936579109s) [upload=12022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:47.347237969 (11h31m46.936582956s) [upload=12022]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:11:47.802049075 (11h31m47.391394053s) [seg_pos={0 0}, upload=12022]: Begin upload segment... Aug 7 21:11:47.802118007 (11h31m47.391462994s) [seg_pos={0 0}, upload=12022]: Done begin upload segment. Aug 7 21:11:47.802142414 (11h31m47.391487401s) [noise=true , address=104.36.82.22:4541 , piece_id=Q5X72MXPD2VX3TRLK37VB3OV5Y4WLD4P6CWPAVL2X4D6BSIIOFSQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802186428 (11h31m47.391531415s) [noise=true , address=104.36.82.5:4846 , piece_id=6NNDPTLCBYLGFWU2H57AIRLEEXTREPMET5R2YZETWQYPIP2ZUHHQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802550154 (11h31m47.391895141s) [noise=true , address=104.36.82.46:4501 , piece_id=CVPKLEXG5FMF3NAR3ONF7LGIWUXJ7OSF5BBOTJ3BLQE4CFFNQKBA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802702837 (11h31m47.392047814s) [noise=true , address=45.91.116.9:4268 , piece_id=VJP2ZZ3UMB2XMXGDGIV45YOLKHRNNJORX2NIB3SHBCI4UZNJEN7A, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802844608 (11h31m47.392189595s) [noise=true , address=104.36.82.16:4328 , piece_id=BFL3YNCSEKYNXJV6PJDPERHGUC6PDQZ52JNSLZDZPH3DCLYCAJPA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803081390 (11h31m47.392426377s) [noise=true , address=104.36.82.47:4523 , piece_id=4WH6OVD7YSUMLPUBHAQPDR5VYX7Y36WKS3MSM7ULY6CI73BALULQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803116328 (11h31m47.392461315s) [noise=true , address=45.91.116.11:4150 , piece_id=64KLQELYVF4SAJJYCFSQD3X5USFFYLXGNOJOQGZ36LOYOFEJ2UBQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803402735 (11h31m47.392747722s) [noise=true , address=104.36.82.12:4165 , piece_id=BBVGOIQK3IBDXSA5KXGHV4Z4TRPZYJQGJT3PYQJXZUI7ZJ55KRJA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803414909 (11h31m47.392759896s) [noise=true , address=104.36.82.39:4752 , piece_id=ITMK3WWPOHUBVDUV2ZUL65XLBGCIVZKGUD544R2K3FCAMTQN5TXQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803693041 (11h31m47.393038028s) [noise=true , address=104.36.82.36:4622 , piece_id=JYCNDCLZYJWGEB4ZWJM5ES3G25WPLBPOY6UGXI37SFC467KTIZEA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803929323 (11h31m47.39327431s) [noise=true , address=104.36.82.42:4813 , piece_id=UAXL4ZUQ4W6P3DLPKIWW7KC3U2MZPZQ25ZA7Z4A7BFQ42EMJZAXQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804163892 (11h31m47.393508879s) [noise=true , address=104.36.82.58:4269 , piece_id=QX6CM6DHSIW2KXY2Z6NRMSRG3HMBRNBQ3JRPHXBGOQMEL5HVZ5ZQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804383622 (11h31m47.393728609s) [noise=true , address=45.91.116.4:4969 , piece_id=GVQDU7BRCLA6PHFSRGOJZBAV6QZQRI2ZEG3EM6I2TJAZ4JAE4P7Q, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804621557 (11h31m47.393966544s) [noise=true , address=104.36.82.23:4453 , piece_id=BLRIGJ5Z3NNUPUHC2YZW4PDMK2SDUVPE75GO5T4WSFTT2FDOEEOQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804837270 (11h31m47.394182257s) [noise=true , address=104.36.82.2:4744 , piece_id=V6TRPHHDLGWKVLQH7BT6AVYVOKVVP3EHL67D62SPHAKNMWICYJWQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804911372 (11h31m47.394256359s) [noise=true , address=104.36.82.43:4849 , piece_id=IZH45DQEPVT7WEQ2UDAAA7OS37XKMKG4VBJRRUEKQ3G36YUM2LMQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.805071538 (11h31m47.394416515s) [noise=true , address=64.49.13.170:4728 , piece_id=LED5VGYENQQBGOP3UH4NKT6O6TH4M6OEZCIXZ5N7AYWFLW452KSA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.805261793 (11h31m47.39460677s) [noise=true , address=104.36.82.44:4899 , piece_id=I5PVQ6WHWORI2TFF2SJSZDAUDH3FX5DVNPP23VC6JECMDAO6ZM6Q, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:48.969580099 (11h31m48.558925086s) [upload=12022]: Next returned nil segment Aug 7 21:11:48.969596991 (11h31m48.558941978s) [upload=12022]: Waiting for error group managing segments... Aug 7 21:11:49.265222255 (11h31m48.854567242s) [noise=true , address=64.49.13.170:4728 , piece_id=LED5VGYENQQBGOP3UH4NKT6O6TH4M6OEZCIXZ5N7AYWFLW452KSA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.737280759 (11h31m49.326625746s) [noise=true , address=104.36.82.58:4269 , piece_id=QX6CM6DHSIW2KXY2Z6NRMSRG3HMBRNBQ3JRPHXBGOQMEL5HVZ5ZQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.753249862 (11h31m49.342594849s) [noise=true , address=104.36.82.12:4165 , piece_id=BBVGOIQK3IBDXSA5KXGHV4Z4TRPZYJQGJT3PYQJXZUI7ZJ55KRJA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.780910703 (11h31m49.37025569s) [noise=true , address=45.91.116.9:4268 , piece_id=VJP2ZZ3UMB2XMXGDGIV45YOLKHRNNJORX2NIB3SHBCI4UZNJEN7A, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.813098015 (11h31m49.402442992s) [noise=true , address=104.36.82.46:4501 , piece_id=CVPKLEXG5FMF3NAR3ONF7LGIWUXJ7OSF5BBOTJ3BLQE4CFFNQKBA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.827616893 (11h31m49.41696188s) [noise=true , address=104.36.82.42:4813 , piece_id=UAXL4ZUQ4W6P3DLPKIWW7KC3U2MZPZQ25ZA7Z4A7BFQ42EMJZAXQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.844644011 (11h31m49.433988998s) [noise=true , address=104.36.82.5:4846 , piece_id=6NNDPTLCBYLGFWU2H57AIRLEEXTREPMET5R2YZETWQYPIP2ZUHHQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.896979020 (11h31m49.486324007s) [noise=true , address=104.36.82.16:4328 , piece_id=BFL3YNCSEKYNXJV6PJDPERHGUC6PDQZ52JNSLZDZPH3DCLYCAJPA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.913302922 (11h31m49.502647909s) [noise=true , address=104.36.82.36:4622 , piece_id=JYCNDCLZYJWGEB4ZWJM5ES3G25WPLBPOY6UGXI37SFC467KTIZEA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.928574831 (11h31m49.517919809s) [noise=true , address=104.36.82.22:4541 , piece_id=Q5X72MXPD2VX3TRLK37VB3OV5Y4WLD4P6CWPAVL2X4D6BSIIOFSQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.967608171 (11h31m49.556953148s) [noise=true , address=104.36.82.43:4849 , piece_id=IZH45DQEPVT7WEQ2UDAAA7OS37XKMKG4VBJRRUEKQ3G36YUM2LMQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.087000568 (11h31m49.676345555s) [noise=true , address=45.91.116.11:4150 , piece_id=64KLQELYVF4SAJJYCFSQD3X5USFFYLXGNOJOQGZ36LOYOFEJ2UBQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.098037264 (11h31m49.687382241s) [noise=true , address=45.91.116.4:4969 , piece_id=GVQDU7BRCLA6PHFSRGOJZBAV6QZQRI2ZEG3EM6I2TJAZ4JAE4P7Q, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.160148408 (11h31m49.749493395s) [noise=true , address=104.36.82.47:4523 , piece_id=4WH6OVD7YSUMLPUBHAQPDR5VYX7Y36WKS3MSM7ULY6CI73BALULQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.160172064 (11h31m49.749517051s) [seg_pos={0 0}, upload=12022]: Segment reached optimal threshold of 14 pieces. Aug 7 21:11:50.160367668 (11h31m49.749712655s) [noise=true , address=104.36.82.44:4899 , piece_id=I5PVQ6WHWORI2TFF2SJSZDAUDH3FX5DVNPP23VC6JECMDAO6ZM6Q, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.160428794 (11h31m49.749773782s) [noise=true , address=104.36.82.23:4453 , piece_id=BLRIGJ5Z3NNUPUHC2YZW4PDMK2SDUVPE75GO5T4WSFTT2FDOEEOQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.160601615 (11h31m49.749946592s) [noise=true , address=104.36.82.2:4744 , piece_id=V6TRPHHDLGWKVLQH7BT6AVYVOKVVP3EHL67D62SPHAKNMWICYJWQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: read tcp 188.166.225.170:55148->104.36.82.2:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:11:50.160658343 (11h31m49.75000333s) [noise=true , address=104.36.82.39:4752 , piece_id=ITMK3WWPOHUBVDUV2ZUL65XLBGCIVZKGUD544R2K3FCAMTQN5TXQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: read tcp 188.166.225.170:46618->104.36.82.39:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:11:50.160688220 (11h31m49.750033207s) [seg_pos={0 0}, upload=12022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:11:50.160746132 (11h31m49.750091179s) [upload=12022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:11:50.160752644 (11h31m49.750097651s) [upload=12022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:50.684435488 (11h31m50.273780475s) [upload=12022]: Done uploading segments... Aug 7 21:11:50.684472599 (11h31m50.273817576s) [upload=12022]: Upload finished. err: <nil> Duration: 6.765251661s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................X..XXX] duration: 30.001071342s FAIL (2025-08-07 21:16:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 567253970 (11h33m16.156598957s) [noise=true , address=45.91.116.5:4298 , piece_id=Y4FCHPKGJXDGWLH2FT6M5PJV5RA4UOYFQ3SHTWFJN4WO3FTFFG2A, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.568348233 (11h33m16.15769322s) [noise=true , address=45.91.116.12:4226 , piece_id=5QFSWOPQLZO2KEP4J27RDULUD4VCQSIE6BZ6BO3MFVFAP67OMW7A, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.570326768 (11h33m16.159671755s) [noise=true , address=77.90.31.11:4553 , piece_id=4QVTWAAY3JK3EQJIHK7ANW6E4UYUI3KXCHZOWRK75JHZZRPZ6ENQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.571062165 (11h33m16.160407143s) [noise=true , address=45.91.116.11:4119 , piece_id=6PDH442NIXRNQCIFX2Y4PNKU5C73T2H6KNHKGMIW2UGCTC5NX7ZA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.572859514 (11h33m16.162204501s) [noise=true , address=104.36.82.15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573225655 (11h33m16.162570642s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573527402 (11h33m16.162872389s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573918310 (11h33m16.163263297s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574198737 (11h33m16.163543724s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574518488 (11h33m16.163863465s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574831648 (11h33m16.164176635s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:18.308274980 (11h33m17.897619967s) [upload=12046]: Next returned nil segment Aug 7 21:13:18.308295720 (11h33m17.897640707s) [upload=12046]: Waiting for error group managing segments... Aug 7 21:13:18.656166252 (11h33m18.245511229s) [noise=true , address=77.90.31.13:4626 , piece_id=S4OKGN6SKZ7GLYKCT3S4QSQFK75FX3KAXGQAOORJRVDJ7HV5UTWQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.668187531 (11h33m18.257532518s) [noise=true , address=104.36.82.6:4873 , piece_id=VLAKUO36OHVGPY3Q723O5OUKURJ2SWREZC3SABUX2J7O6MBM7Y2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.721769396 (11h33m18.311114384s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.741305974 (11h33m18.330650951s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.780529499 (11h33m18.369874486s) [noise=true , address=77.90.31.11:4553 , piece_id=4QVTWAAY3JK3EQJIHK7ANW6E4UYUI3KXCHZOWRK75JHZZRPZ6ENQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.865633524 (11h33m18.454978511s) [noise=true , address=45.91.116.11:4119 , piece_id=6PDH442NIXRNQCIFX2Y4PNKU5C73T2H6KNHKGMIW2UGCTC5NX7ZA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880648330 (11h33m18.469993317s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880718855 (11h33m18.470063842s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.935042530 (11h33m18.524387517s) [noise=true , address=45.91.116.5:4298 , piece_id=Y4FCHPKGJXDGWLH2FT6M5PJV5RA4UOYFQ3SHTWFJN4WO3FTFFG2A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.969979703 (11h33m18.55932468s) [noise=true , address=45.91.116.5:4299 , piece_id=5YQ5OJAPBA3N2KMYSVWQCNOTH7QPX2WCHHUXTLNSVNH2P7MZEJ3A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.972928735 (11h33m18.562273722s) [noise=true , address=45.91.116.12:4226 , piece_id=5QFSWOPQLZO2KEP4J27RDULUD4VCQSIE6BZ6BO3MFVFAP67OMW7A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.047677267 (11h33m18.637022254s) [noise=true , address=104.36.82.15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082384121 (11h33m18.671729108s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082409660 (11h33m18.671754647s) [seg_pos={0 0}, upload=12046]: Segment reached optimal threshold of 13 pieces. Aug 7 21:13:19.082628518 (11h33m18.671973495s) [noise=true , address=77.90.31.14:4661 , piece_id=P4DQFG2ACQKL52JQZFCTIFCUHFPLDIYEOOG4OPBJY4MIXBVKCYDQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Aug 7 21:13:19.082795017 (11h33m18.672139994s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: read tcp 188.166.225.170:56758->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082822870 (11h33m18.672167857s) [noise=true , address=104.36.82.16:4294 , piece_id=QN2G4ZPHQ2VBVMEJFPY3OLOGLHZRKEFC4SXOXSHFBQMD47QMAM7Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:47752->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082839501 (11h33m18.672184478s) [seg_pos={0 0}, upload=12046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:13:19.082910688 (11h33m18.672255675s) [upload=12046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:19.082914756 (11h33m18.672259743s) [upload=12046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:19.387715025 (11h33m18.977060012s) [upload=12046]: Done uploading segments... Aug 7 21:13:19.387739612 (11h33m18.977084599s) [upload=12046]: Upload finished. err: <nil> Aug 7 21:13:45.069592425 (11h33m44.658937412s) [upload=12058]: Starting upload Aug 7 21:13:45.069646459 (11h33m44.658991446s) [upload=12058]: Done starting upload Aug 7 21:13:45.069729608 (11h33m44.659074595s) [upload=12058]: Uploading segments... Aug 7 21:13:45.069894263 (11h33m44.65923925s) [upload=12058]: Got next segment. Inline: false Aug 7 21:13:45.069927166 (11h33m44.659272153s) [upload=12058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.069931384 (11h33m44.659276371s) [upload=12058]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:14.428211569 (11h34m14.017556556s) [upload=12058]: Next segment err: context deadline exceeded Aug 7 21:14:14.428287775 (11h34m14.017632762s) [upload=12058]: Done uploading segments... Aug 7 21:14:14.428338853 (11h34m14.01768384s) [upload=12058]: Upload finished. err: context deadline exceeded Aug 7 21:14:49.077304183 (11h34m48.66664917s) [upload=12074]: Starting upload Aug 7 21:14:49.077411358 (11h34m48.666756365s) [upload=12074]: Done starting upload Aug 7 21:14:49.077536547 (11h34m48.666881534s) [upload=12074]: Uploading segments... Aug 7 21:14:49.077698226 (11h34m48.667043214s) [upload=12074]: Got next segment. Inline: false Aug 7 21:14:49.077743253 (11h34m48.66708824s) [upload=12074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:49.077748543 (11h34m48.66709353s) [upload=12074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:18.426190970 (11h35m18.015535997s) [upload=12074]: Next segment err: context deadline exceeded Aug 7 21:15:18.426350985 (11h35m18.015695972s) [upload=12074]: Done uploading segments... Aug 7 21:15:18.426379730 (11h35m18.015724707s) [upload=12074]: Upload finished. err: context deadline exceeded Aug 7 21:15:36.581505585 (11h35m36.170850572s) [upload=12086]: Starting upload Aug 7 21:15:36.581560250 (11h35m36.170905227s) [upload=12086]: Done starting upload Aug 7 21:15:36.581682133 (11h35m36.17102711s) [upload=12086]: Uploading segments... Aug 7 21:15:36.581769470 (11h35m36.171114457s) [upload=12086]: Got next segment. Inline: false Aug 7 21:15:36.581824996 (11h35m36.171169983s) [upload=12086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:36.581829786 (11h35m36.171174773s) [upload=12086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:05.932290909 (11h36m5.521635897s) [upload=12086]: Next segment err: context deadline exceeded Aug 7 21:16:05.932368198 (11h36m5.521713185s) [upload=12086]: Done uploading segments... Aug 7 21:16:05.932396572 (11h36m5.521741559s) [upload=12086]: Upload finished. err: context deadline exceeded Duration: 30.001083615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................X..XX] duration: 30.001026736s FAIL (2025-08-07 21:16:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001046473s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.000638952s FAIL (2025-08-07 21:16:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00065857s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000295413s FAIL (2025-08-07 21:15:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000323397s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000906223s FAIL (2025-08-07 21:15:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00093165s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.001382765s FAIL (2025-08-07 21:15:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== hed. err: <nil> Aug 7 21:09:25.498062703 (11h29m25.08740769s) [upload=11978]: Starting upload Aug 7 21:09:25.498115193 (11h29m25.08746018s) [upload=11978]: Done starting upload Aug 7 21:09:25.498162373 (11h29m25.08750736s) [upload=11978]: Uploading segments... Aug 7 21:09:25.498190667 (11h29m25.087535654s) [upload=11978]: Got next segment. Inline: true Aug 7 21:09:25.498196007 (11h29m25.087540994s) [upload=11978]: Waiting for error group managing segments... Aug 7 21:09:25.498229983 (11h29m25.08757497s) [upload=11978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:25.498233650 (11h29m25.087578637s) [upload=11978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:25.498236134 (11h29m25.087581121s) [upload=11978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:25.808420372 (11h29m25.397765359s) [upload=11978]: Done uploading segments... Aug 7 21:09:25.808436974 (11h29m25.397781961s) [upload=11978]: Upload finished. err: <nil> Aug 7 21:09:47.139455442 (11h29m46.728800429s) [upload=11988]: Starting upload Aug 7 21:09:47.139503314 (11h29m46.728848301s) [upload=11988]: Done starting upload Aug 7 21:09:47.139548831 (11h29m46.728893818s) [upload=11988]: Uploading segments... Aug 7 21:09:47.139579450 (11h29m46.728924437s) [upload=11988]: Got next segment. Inline: true Aug 7 21:09:47.139583277 (11h29m46.728928264s) [upload=11988]: Waiting for error group managing segments... Aug 7 21:09:47.139618274 (11h29m46.728963261s) [upload=11988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:47.139622451 (11h29m46.728967439s) [upload=11988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:47.139624797 (11h29m46.728969784s) [upload=11988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.434297548 (11h29m47.023642535s) [upload=11988]: Done uploading segments... Aug 7 21:09:47.434326894 (11h29m47.023671872s) [upload=11988]: Upload finished. err: <nil> Aug 7 21:10:17.433462616 (11h30m17.022807593s) [upload=11995]: Starting upload Aug 7 21:10:17.433520777 (11h30m17.022865764s) [upload=11995]: Done starting upload Aug 7 21:10:17.433584859 (11h30m17.022929846s) [upload=11995]: Uploading segments... Aug 7 21:10:17.433692035 (11h30m17.023037012s) [upload=11995]: Got next segment. Inline: true Aug 7 21:10:17.433696885 (11h30m17.023041872s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:10:17.433744876 (11h30m17.023089853s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:17.433749755 (11h30m17.023094742s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:17.433754204 (11h30m17.023099191s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:17.710858662 (11h30m17.300203709s) [upload=11995]: Done uploading segments... Aug 7 21:10:17.710881997 (11h30m17.300227024s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:10:36.218517401 (11h30m35.807862378s) [upload=11998]: Starting upload Aug 7 21:10:36.218649954 (11h30m35.807994941s) [upload=11998]: Done starting upload Aug 7 21:10:36.218713896 (11h30m35.808058913s) [upload=11998]: Uploading segments... Aug 7 21:10:36.218766106 (11h30m35.808111103s) [upload=11998]: Got next segment. Inline: true Aug 7 21:10:36.218772398 (11h30m35.808117425s) [upload=11998]: Waiting for error group managing segments... Aug 7 21:10:36.218820571 (11h30m35.808165558s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:36.218825550 (11h30m35.808170537s) [upload=11998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:36.218828946 (11h30m35.808173943s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:36.494741534 (11h30m36.084086521s) [upload=11998]: Done uploading segments... Aug 7 21:10:36.494760030 (11h30m36.084105007s) [upload=11998]: Upload finished. err: <nil> Aug 7 21:11:36.918559943 (11h31m36.50790493s) [upload=12018]: Starting upload Aug 7 21:11:36.918615830 (11h31m36.507960817s) [upload=12018]: Done starting upload Aug 7 21:11:36.918660295 (11h31m36.508005282s) [upload=12018]: Uploading segments... Aug 7 21:11:36.918688629 (11h31m36.508033606s) [upload=12018]: Got next segment. Inline: true Aug 7 21:11:36.918691745 (11h31m36.508036732s) [upload=12018]: Waiting for error group managing segments... Aug 7 21:11:36.918721552 (11h31m36.508066539s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:36.918725530 (11h31m36.508070517s) [upload=12018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:36.918727794 (11h31m36.508072781s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:37.211063805 (11h31m36.800408792s) [upload=12018]: Done uploading segments... Aug 7 21:11:37.211087029 (11h31m36.800432026s) [upload=12018]: Upload finished. err: <nil> Aug 7 21:12:15.260577538 (11h32m14.849922525s) [upload=12032]: Starting upload Aug 7 21:12:15.260632023 (11h32m14.84997701s) [upload=12032]: Done starting upload Aug 7 21:12:15.260678571 (11h32m14.850023548s) [upload=12032]: Uploading segments... Aug 7 21:12:15.260706434 (11h32m14.850051421s) [upload=12032]: Got next segment. Inline: true Aug 7 21:12:15.260710011 (11h32m14.850054998s) [upload=12032]: Waiting for error group managing segments... Aug 7 21:12:15.260743897 (11h32m14.850088874s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:15.260747644 (11h32m14.850092631s) [upload=12032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:15.260750389 (11h32m14.850095376s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:15.565323071 (11h32m15.154668058s) [upload=12032]: Done uploading segments... Aug 7 21:12:15.565341296 (11h32m15.154686283s) [upload=12032]: Upload finished. err: <nil> Aug 7 21:12:41.001560374 (11h32m40.590905361s) [upload=12034]: Starting upload Aug 7 21:12:41.001659143 (11h32m40.59100413s) [upload=12034]: Done starting upload Aug 7 21:12:41.001745669 (11h32m40.591090666s) [upload=12034]: Uploading segments... Aug 7 21:12:41.001852704 (11h32m40.591197701s) [upload=12034]: Got next segment. Inline: true Aug 7 21:12:41.001857372 (11h32m40.591202359s) [upload=12034]: Waiting for error group managing segments... Aug 7 21:12:41.001907849 (11h32m40.591252836s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:41.001913169 (11h32m40.591258156s) [upload=12034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:41.001915584 (11h32m40.591260571s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:41.286244505 (11h32m40.875589493s) [upload=12034]: Done uploading segments... Aug 7 21:12:41.286260567 (11h32m40.875605544s) [upload=12034]: Upload finished. err: <nil> Aug 7 21:13:16.624584146 (11h33m16.213929123s) [upload=12047]: Starting upload Aug 7 21:13:16.624659630 (11h33m16.214004607s) [upload=12047]: Done starting upload Aug 7 21:13:16.624710127 (11h33m16.214055114s) [upload=12047]: Uploading segments... Aug 7 21:13:16.624745715 (11h33m16.214090702s) [upload=12047]: Got next segment. Inline: true Aug 7 21:13:16.624750654 (11h33m16.214095641s) [upload=12047]: Waiting for error group managing segments... Aug 7 21:13:16.624782977 (11h33m16.214127964s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:16.624786804 (11h33m16.214131791s) [upload=12047]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:16.624789339 (11h33m16.214134326s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.974391598 (11h33m45.563736596s) [upload=12047]: Done uploading segments... Aug 7 21:13:45.974419923 (11h33m45.56376491s) [upload=12047]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:31.328315035 (11h34m30.917660012s) [upload=12067]: Starting upload Aug 7 21:14:31.328377385 (11h34m30.917722373s) [upload=12067]: Done starting upload Aug 7 21:14:31.328450665 (11h34m30.917795652s) [upload=12067]: Uploading segments... Aug 7 21:14:31.328587126 (11h34m30.917932113s) [upload=12067]: Got next segment. Inline: true Aug 7 21:14:31.328592336 (11h34m30.917937323s) [upload=12067]: Waiting for error group managing segments... Aug 7 21:14:31.328654706 (11h34m30.917999693s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:31.328660026 (11h34m30.918005013s) [upload=12067]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:31.328662631 (11h34m30.918007618s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:00.687780769 (11h35m0.277125756s) [upload=12067]: Done uploading segments... Aug 7 21:15:00.687819252 (11h35m0.27716424s) [upload=12067]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:17.371028003 (11h35m16.96037298s) [upload=12080]: Starting upload Aug 7 21:15:17.371083610 (11h35m16.960428587s) [upload=12080]: Done starting upload Aug 7 21:15:17.371134738 (11h35m16.960479725s) [upload=12080]: Uploading segments... Aug 7 21:15:17.371166228 (11h35m16.960511215s) [upload=12080]: Got next segment. Inline: true Aug 7 21:15:17.371170035 (11h35m16.960515022s) [upload=12080]: Waiting for error group managing segments... Aug 7 21:15:17.371203299 (11h35m16.960548286s) [upload=12080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:17.371207227 (11h35m16.960552214s) [upload=12080]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:17.371209821 (11h35m16.960554809s) [upload=12080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:46.717162709 (11h35m46.306507686s) [upload=12080]: Done uploading segments... Aug 7 21:15:46.717181444 (11h35m46.306526431s) [upload=12080]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001413324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.00057288s FAIL (2025-08-07 21:15:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 2039]: Uploading piece... Aug 7 21:12:52.233595037 (11h32m51.822940024s) [noise=true , address=104.36.82.43:4866 , piece_id=AO5FAMAICZ2WFNV3L75ULE6UDCKGENWEBO47HXEA5ZVZ6KPOBFXQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233603493 (11h32m51.82294848s) [noise=true , address=104.36.82.18:4378 , piece_id=ZJJ542SCBSUKTG7QB5PQFTDRPHMFHN32RY6LXNSDLU7X6SCEVBNQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233891473 (11h32m51.82323646s) [noise=true , address=104.36.82.56:4247 , piece_id=X7EBBRNI7ZUNCR6SYWNS3TTBBQ4Z3BZZYWZNBYFBYRWYTV4TU5VQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233989401 (11h32m51.823334388s) [noise=true , address=45.91.116.8:4067 , piece_id=GYAFL34K7CTRS3SOTQZFERYUQGQKVOBWUTPJ3E5CFYBZ7ON5RQVA, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234150710 (11h32m51.823495697s) [noise=true , address=45.91.116.7:4034 , piece_id=RDFD2PLXXJ3GRAOR5ZLPKDLRK5IGJTZSBSNVG4A6XFZB75WPJ22Q, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234254518 (11h32m51.823599506s) [noise=true , address=104.36.82.44:4882 , piece_id=M7WROH24EKE2EMGFKDDDS6UOM3E5P4OXLT2QDIBXN2BCYFZCYUMA, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234402521 (11h32m51.823747508s) [noise=true , address=104.36.82.62:4427 , piece_id=QTWU24TIM66G6SQXGSU4H22JZTWDPYSZVIZV4FGRILO77SGECVNQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234520336 (11h32m51.823865313s) [noise=true , address=104.36.82.60:4348 , piece_id=KYGK6JIIPPPB3WXEJME7RHT4YU2K5YA6XO2UFS7RVRFC2MK5HL3Q, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234683298 (11h32m51.824028285s) [noise=true , address=104.36.82.41:4772 , piece_id=BGC5NOQVRQ4BQEUPDI5UDRVKUBSTZMOQKGKLTHLTBNWFTORUWZQQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234718776 (11h32m51.824063753s) [noise=true , address=104.36.82.34:4571 , piece_id=APKT66JAXIQNA5UXXM3FCX7JCN5ARKXPDUD6ZJ6WB46MMR2VF3UQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234915192 (11h32m51.824260179s) [noise=true , address=45.91.116.5:4325 , piece_id=P5KGQGDJXUKBS7F7ZK6TIP52U7I2OI74XFJTQSSQRFA2AS4WQDVQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.235686748 (11h32m51.825031735s) [noise=true , address=104.36.82.59:4333 , piece_id=BR6IGKYFNUAH2ZWKQMUQHYGMOL27HFFGN6PEPJOP7WODCYOVUJEQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:53.755589114 (11h32m53.344934101s) [upload=12039]: Next returned nil segment Aug 7 21:12:53.755603641 (11h32m53.344948629s) [upload=12039]: Waiting for error group managing segments... Aug 7 21:12:54.242895872 (11h32m53.832240859s) [noise=true , address=104.36.82.44:4882 , piece_id=M7WROH24EKE2EMGFKDDDS6UOM3E5P4OXLT2QDIBXN2BCYFZCYUMA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.247833398 (11h32m53.837178375s) [noise=true , address=104.36.82.23:4453 , piece_id=OIB2N3NXKLOBJS2UV353TMJMCMK3QM4UGLNKX7P6PUZDT2MRNILQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.248473462 (11h32m53.837818449s) [noise=true , address=104.36.82.16:4300 , piece_id=RVRXMZLRANZJMWSQAF7HTJUULYADEA2I6GPHDTFGKNS6FNWE36VA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.248750062 (11h32m53.83809505s) [noise=true , address=104.36.82.60:4348 , piece_id=KYGK6JIIPPPB3WXEJME7RHT4YU2K5YA6XO2UFS7RVRFC2MK5HL3Q, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.249467144 (11h32m53.838812131s) [noise=true , address=104.36.82.18:4378 , piece_id=ZJJ542SCBSUKTG7QB5PQFTDRPHMFHN32RY6LXNSDLU7X6SCEVBNQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.269168297 (11h32m53.858513304s) [noise=true , address=104.36.82.56:4247 , piece_id=X7EBBRNI7ZUNCR6SYWNS3TTBBQ4Z3BZZYWZNBYFBYRWYTV4TU5VQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.286754543 (11h32m53.87609953s) [noise=true , address=104.36.82.34:4571 , piece_id=APKT66JAXIQNA5UXXM3FCX7JCN5ARKXPDUD6ZJ6WB46MMR2VF3UQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.315127418 (11h32m53.904472395s) [noise=true , address=104.36.82.43:4866 , piece_id=AO5FAMAICZ2WFNV3L75ULE6UDCKGENWEBO47HXEA5ZVZ6KPOBFXQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.339819320 (11h32m53.929164307s) [noise=true , address=104.36.82.55:4195 , piece_id=CKLDMZFNPKVSRXP5A6Z7DIRQOTMLENLOL7XBDWJTMIRDNCFV3ZIA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.352457851 (11h32m53.941802858s) [noise=true , address=104.36.82.59:4333 , piece_id=BR6IGKYFNUAH2ZWKQMUQHYGMOL27HFFGN6PEPJOP7WODCYOVUJEQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.366096104 (11h32m53.955441091s) [noise=true , address=104.36.82.6:4890 , piece_id=DBUDIACR7PEFAFGSOBGEIMWWYDQ4QD4ZWFQ5MNIT2DRDSHWJUE7A, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.366979544 (11h32m53.956324531s) [noise=true , address=104.36.82.62:4427 , piece_id=QTWU24TIM66G6SQXGSU4H22JZTWDPYSZVIZV4FGRILO77SGECVNQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.381141350 (11h32m53.970486347s) [noise=true , address=104.36.82.41:4772 , piece_id=BGC5NOQVRQ4BQEUPDI5UDRVKUBSTZMOQKGKLTHLTBNWFTORUWZQQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.391573819 (11h32m53.980918806s) [noise=true , address=45.91.116.8:4067 , piece_id=GYAFL34K7CTRS3SOTQZFERYUQGQKVOBWUTPJ3E5CFYBZ7ON5RQVA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.391594779 (11h32m53.980939766s) [seg_pos={0 0}, upload=12039]: Segment reached optimal threshold of 14 pieces. Aug 7 21:12:54.391723245 (11h32m53.981068232s) [noise=true , address=45.91.116.7:4034 , piece_id=RDFD2PLXXJ3GRAOR5ZLPKDLRK5IGJTZSBSNVG4A6XFZB75WPJ22Q, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391809851 (11h32m53.981154838s) [noise=true , address=45.91.116.5:4325 , piece_id=P5KGQGDJXUKBS7F7ZK6TIP52U7I2OI74XFJTQSSQRFA2AS4WQDVQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391822284 (11h32m53.981167271s) [noise=true , address=104.36.82.45:4939 , piece_id=UIVN6HS5ZIB44VIN3PBQSMVEEFAYA3DF43KAALY6G6HCFADLZWIQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection; write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection Aug 7 21:12:54.391862171 (11h32m53.981207158s) [noise=true , address=104.36.82.43:4867 , piece_id=ZZCKCBXHV7POWCGCQ7DAJ2QE5A4UYFEERTSCUGVWU7UC76BYZTYA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zENB6bYmb9hqtGTcJ1afKVbcNtbEsGo4dpsSEdh2feqHifmHQ): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391878141 (11h32m53.981223118s) [seg_pos={0 0}, upload=12039]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:12:54.391977522 (11h32m53.981322509s) [upload=12039]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:12:54.391981659 (11h32m53.981326646s) [upload=12039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.855789778 (11h32m54.445134765s) [upload=12039]: Done uploading segments... Aug 7 21:12:54.855808144 (11h32m54.445153121s) [upload=12039]: Upload finished. err: <nil> Aug 7 21:13:45.303666216 (11h33m44.893011253s) [upload=12059]: Starting upload Aug 7 21:13:45.303786125 (11h33m44.893131112s) [upload=12059]: Done starting upload Aug 7 21:13:45.303856099 (11h33m44.893201126s) [upload=12059]: Uploading segments... Aug 7 21:13:45.304260994 (11h33m44.893605981s) [upload=12059]: Got next segment. Inline: false Aug 7 21:13:45.304300049 (11h33m44.893645036s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.304305609 (11h33m44.893650596s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:44.664782364 (11h34m44.254127351s) [upload=12059]: Next segment err: context deadline exceeded Aug 7 21:14:44.664969041 (11h34m44.254314028s) [upload=12059]: Done uploading segments... Aug 7 21:14:44.665004269 (11h34m44.254349256s) [upload=12059]: Upload finished. err: context deadline exceeded Aug 7 21:14:45.557957097 (11h34m45.147302084s) [upload=12072]: Starting upload Aug 7 21:14:45.558041418 (11h34m45.147386395s) [upload=12072]: Done starting upload Aug 7 21:14:45.558153183 (11h34m45.14749817s) [upload=12072]: Uploading segments... Aug 7 21:14:45.558236202 (11h34m45.147581189s) [upload=12072]: Got next segment. Inline: false Aug 7 21:14:45.558263704 (11h34m45.147608691s) [upload=12072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.558268123 (11h34m45.1476131s) [upload=12072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:44.909553125 (11h35m44.498898112s) [upload=12072]: Next segment err: context deadline exceeded Aug 7 21:15:44.909693113 (11h35m44.49903809s) [upload=12072]: Done uploading segments... Aug 7 21:15:44.909705847 (11h35m44.499050834s) [upload=12072]: Upload finished. err: context deadline exceeded Duration: 1m0.000586155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................................X.XX] duration: 1m0.000870232s FAIL (2025-08-07 21:15:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== shed. err: <nil> Aug 7 21:07:37.321661864 (11h27m36.911006841s) [upload=11949]: Starting upload Aug 7 21:07:37.321718112 (11h27m36.911063099s) [upload=11949]: Done starting upload Aug 7 21:07:37.321771434 (11h27m36.911116411s) [upload=11949]: Uploading segments... Aug 7 21:07:37.321872968 (11h27m36.911217945s) [upload=11949]: Got next segment. Inline: true Aug 7 21:07:37.321877877 (11h27m36.911222865s) [upload=11949]: Waiting for error group managing segments... Aug 7 21:07:37.321923275 (11h27m36.911268262s) [upload=11949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:07:37.321928314 (11h27m36.911273301s) [upload=11949]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:07:37.321930418 (11h27m36.911275405s) [upload=11949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:07:37.599927314 (11h27m37.189272301s) [upload=11949]: Done uploading segments... Aug 7 21:07:37.599948635 (11h27m37.189293672s) [upload=11949]: Upload finished. err: <nil> Aug 7 21:08:35.851762080 (11h28m35.441107057s) [upload=11967]: Starting upload Aug 7 21:08:35.851828216 (11h28m35.441173203s) [upload=11967]: Done starting upload Aug 7 21:08:35.851888952 (11h28m35.441233939s) [upload=11967]: Uploading segments... Aug 7 21:08:35.851977722 (11h28m35.441322709s) [upload=11967]: Got next segment. Inline: true Aug 7 21:08:35.851983424 (11h28m35.441328411s) [upload=11967]: Waiting for error group managing segments... Aug 7 21:08:35.852017519 (11h28m35.441362506s) [upload=11967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:08:35.852021105 (11h28m35.441366093s) [upload=11967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:08:35.852023460 (11h28m35.441368447s) [upload=11967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:08:36.115286628 (11h28m35.704631615s) [upload=11967]: Done uploading segments... Aug 7 21:08:36.115331112 (11h28m35.704676099s) [upload=11967]: Upload finished. err: <nil> Aug 7 21:09:20.788493181 (11h29m20.377838178s) [upload=11976]: Starting upload Aug 7 21:09:20.788595918 (11h29m20.377940965s) [upload=11976]: Done starting upload Aug 7 21:09:20.788684948 (11h29m20.378029975s) [upload=11976]: Uploading segments... Aug 7 21:09:20.788747948 (11h29m20.378092935s) [upload=11976]: Got next segment. Inline: true Aug 7 21:09:20.788756344 (11h29m20.378101361s) [upload=11976]: Waiting for error group managing segments... Aug 7 21:09:20.788807752 (11h29m20.378152739s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:20.788813985 (11h29m20.378158972s) [upload=11976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:20.788818584 (11h29m20.378163611s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:21.070630248 (11h29m20.659975235s) [upload=11976]: Done uploading segments... Aug 7 21:09:21.070652190 (11h29m20.659997247s) [upload=11976]: Upload finished. err: <nil> Aug 7 21:09:46.948530536 (11h29m46.537875523s) [upload=11987]: Starting upload Aug 7 21:09:46.948588607 (11h29m46.537933584s) [upload=11987]: Done starting upload Aug 7 21:09:46.948642600 (11h29m46.537987587s) [upload=11987]: Uploading segments... Aug 7 21:09:46.948702786 (11h29m46.538047773s) [upload=11987]: Got next segment. Inline: true Aug 7 21:09:46.948707776 (11h29m46.538052763s) [upload=11987]: Waiting for error group managing segments... Aug 7 21:09:46.948747241 (11h29m46.538092228s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:46.948752340 (11h29m46.538097327s) [upload=11987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:46.948754896 (11h29m46.538099883s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.214261274 (11h29m46.803606261s) [upload=11987]: Done uploading segments... Aug 7 21:09:47.214279799 (11h29m46.803624786s) [upload=11987]: Upload finished. err: <nil> Aug 7 21:10:37.572355395 (11h30m37.161700372s) [upload=11999]: Starting upload Aug 7 21:10:37.572410881 (11h30m37.161755868s) [upload=11999]: Done starting upload Aug 7 21:10:37.572467810 (11h30m37.161812797s) [upload=11999]: Uploading segments... Aug 7 21:10:37.572531982 (11h30m37.161876959s) [upload=11999]: Got next segment. Inline: true Aug 7 21:10:37.572536320 (11h30m37.161881307s) [upload=11999]: Waiting for error group managing segments... Aug 7 21:10:37.572570917 (11h30m37.161915904s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:37.572574955 (11h30m37.161919942s) [upload=11999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:37.572577520 (11h30m37.161922507s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:37.838997375 (11h30m37.428342362s) [upload=11999]: Done uploading segments... Aug 7 21:10:37.839014699 (11h30m37.428359686s) [upload=11999]: Upload finished. err: <nil> Aug 7 21:11:31.258043368 (11h31m30.847388355s) [upload=12015]: Starting upload Aug 7 21:11:31.258101790 (11h31m30.847446777s) [upload=12015]: Done starting upload Aug 7 21:11:31.258163838 (11h31m30.847508825s) [upload=12015]: Uploading segments... Aug 7 21:11:31.258195730 (11h31m30.847540717s) [upload=12015]: Got next segment. Inline: true Aug 7 21:11:31.258199567 (11h31m30.847544565s) [upload=12015]: Waiting for error group managing segments... Aug 7 21:11:31.258240175 (11h31m30.847585162s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.258244853 (11h31m30.847589841s) [upload=12015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.258247358 (11h31m30.847592345s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.535852595 (11h31m31.125197582s) [upload=12015]: Done uploading segments... Aug 7 21:11:31.535871041 (11h31m31.125216028s) [upload=12015]: Upload finished. err: <nil> Aug 7 21:11:57.912240700 (11h31m57.501585687s) [upload=12025]: Starting upload Aug 7 21:11:57.912288532 (11h31m57.501633519s) [upload=12025]: Done starting upload Aug 7 21:11:57.912345871 (11h31m57.501690858s) [upload=12025]: Uploading segments... Aug 7 21:11:57.912374877 (11h31m57.501719874s) [upload=12025]: Got next segment. Inline: true Aug 7 21:11:57.912378794 (11h31m57.501723781s) [upload=12025]: Waiting for error group managing segments... Aug 7 21:11:57.912425353 (11h31m57.501770401s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:57.912431234 (11h31m57.501776232s) [upload=12025]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:57.912435933 (11h31m57.50178095s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:57.261749711 (11h32m56.851094698s) [upload=12025]: Done uploading segments... Aug 7 21:12:57.261780350 (11h32m56.851125337s) [upload=12025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:07.178414027 (11h33m6.767759014s) [upload=12044]: Starting upload Aug 7 21:13:07.178467459 (11h33m6.767812446s) [upload=12044]: Done starting upload Aug 7 21:13:07.178514029 (11h33m6.767859016s) [upload=12044]: Uploading segments... Aug 7 21:13:07.178543715 (11h33m6.767888703s) [upload=12044]: Got next segment. Inline: true Aug 7 21:13:07.178547282 (11h33m6.767892259s) [upload=12044]: Waiting for error group managing segments... Aug 7 21:13:07.178580015 (11h33m6.767925002s) [upload=12044]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:07.178583561 (11h33m6.767928548s) [upload=12044]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:07.178585735 (11h33m6.767930722s) [upload=12044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:07.471379642 (11h33m7.060724629s) [upload=12044]: Done uploading segments... Aug 7 21:13:07.471396245 (11h33m7.060741232s) [upload=12044]: Upload finished. err: <nil> Aug 7 21:13:32.020845488 (11h33m31.610190465s) [upload=12052]: Starting upload Aug 7 21:13:32.020909792 (11h33m31.610254769s) [upload=12052]: Done starting upload Aug 7 21:13:32.020999433 (11h33m31.61034443s) [upload=12052]: Uploading segments... Aug 7 21:13:32.021133279 (11h33m31.610478267s) [upload=12052]: Got next segment. Inline: true Aug 7 21:13:32.021141214 (11h33m31.610486231s) [upload=12052]: Waiting for error group managing segments... Aug 7 21:13:32.021222079 (11h33m31.610567066s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:32.021229403 (11h33m31.61057439s) [upload=12052]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:32.021234362 (11h33m31.610579399s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:31.378507697 (11h34m30.967852684s) [upload=12052]: Done uploading segments... Aug 7 21:14:31.378529929 (11h34m30.967874916s) [upload=12052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:36.610398147 (11h34m36.199743124s) [upload=12069]: Starting upload Aug 7 21:14:36.610597218 (11h34m36.199942205s) [upload=12069]: Done starting upload Aug 7 21:14:36.610671039 (11h34m36.200016016s) [upload=12069]: Uploading segments... Aug 7 21:14:36.610800146 (11h34m36.200145133s) [upload=12069]: Got next segment. Inline: true Aug 7 21:14:36.610805957 (11h34m36.200150944s) [upload=12069]: Waiting for error group managing segments... Aug 7 21:14:36.610869649 (11h34m36.200214636s) [upload=12069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:36.610874759 (11h34m36.200219746s) [upload=12069]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:36.610877584 (11h34m36.200222571s) [upload=12069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:35.947737631 (11h35m35.537082618s) [upload=12069]: Done uploading segments... Aug 7 21:15:35.947756376 (11h35m35.537101353s) [upload=12069]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000895379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001169504s FAIL (2025-08-07 21:15:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001190464s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................X..XX] duration: 30.000745893s FAIL (2025-08-07 21:15:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.565371317 (11h33m16.154716304s) [noise=true , address=104.36.82.6:4873 , piece_id=VLAKUO36OHVGPY3Q723O5OUKURJ2SWREZC3SABUX2J7O6MBM7Y2Q, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.565974351 (11h33m16.155319338s) [noise=true , address=45.91.116.5:4299 , piece_id=5YQ5OJAPBA3N2KMYSVWQCNOTH7QPX2WCHHUXTLNSVNH2P7MZEJ3A, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.566321225 (11h33m16.155666212s) [noise=true , address=104.36.82.16:4294 , piece_id=QN2G4ZPHQ2VBVMEJFPY3OLOGLHZRKEFC4SXOXSHFBQMD47QMAM7Q, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.566822494 (11h33m16.156167471s) [noise=true , address=77.90.31.13:4626 , piece_id=S4OKGN6SKZ7GLYKCT3S4QSQFK75FX3KAXGQAOORJRVDJ7HV5UTWQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.567253970 (11h33m16.156598957s) [noise=true , address=45.91.116.5:4298 , piece_id=Y4FCHPKGJXDGWLH2FT6M5PJV5RA4UOYFQ3SHTWFJN4WO3FTFFG2A, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.568348233 (11h33m16.15769322s) [noise=true , address=45.91.116.12:4226 , piece_id=5QFSWOPQLZO2KEP4J27RDULUD4VCQSIE6BZ6BO3MFVFAP67OMW7A, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.570326768 (11h33m16.159671755s) [noise=true , address=77.90.31.11:4553 , piece_id=4QVTWAAY3JK3EQJIHK7ANW6E4UYUI3KXCHZOWRK75JHZZRPZ6ENQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.571062165 (11h33m16.160407143s) [noise=true , address=45.91.116.11:4119 , piece_id=6PDH442NIXRNQCIFX2Y4PNKU5C73T2H6KNHKGMIW2UGCTC5NX7ZA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.572859514 (11h33m16.162204501s) [noise=true , address=104.36.82.15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573225655 (11h33m16.162570642s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573527402 (11h33m16.162872389s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573918310 (11h33m16.163263297s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574198737 (11h33m16.163543724s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574518488 (11h33m16.163863465s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574831648 (11h33m16.164176635s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:18.308274980 (11h33m17.897619967s) [upload=12046]: Next returned nil segment Aug 7 21:13:18.308295720 (11h33m17.897640707s) [upload=12046]: Waiting for error group managing segments... Aug 7 21:13:18.656166252 (11h33m18.245511229s) [noise=true , address=77.90.31.13:4626 , piece_id=S4OKGN6SKZ7GLYKCT3S4QSQFK75FX3KAXGQAOORJRVDJ7HV5UTWQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.668187531 (11h33m18.257532518s) [noise=true , address=104.36.82.6:4873 , piece_id=VLAKUO36OHVGPY3Q723O5OUKURJ2SWREZC3SABUX2J7O6MBM7Y2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.721769396 (11h33m18.311114384s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.741305974 (11h33m18.330650951s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.780529499 (11h33m18.369874486s) [noise=true , address=77.90.31.11:4553 , piece_id=4QVTWAAY3JK3EQJIHK7ANW6E4UYUI3KXCHZOWRK75JHZZRPZ6ENQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.865633524 (11h33m18.454978511s) [noise=true , address=45.91.116.11:4119 , piece_id=6PDH442NIXRNQCIFX2Y4PNKU5C73T2H6KNHKGMIW2UGCTC5NX7ZA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880648330 (11h33m18.469993317s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880718855 (11h33m18.470063842s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.935042530 (11h33m18.524387517s) [noise=true , address=45.91.116.5:4298 , piece_id=Y4FCHPKGJXDGWLH2FT6M5PJV5RA4UOYFQ3SHTWFJN4WO3FTFFG2A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.969979703 (11h33m18.55932468s) [noise=true , address=45.91.116.5:4299 , piece_id=5YQ5OJAPBA3N2KMYSVWQCNOTH7QPX2WCHHUXTLNSVNH2P7MZEJ3A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.972928735 (11h33m18.562273722s) [noise=true , address=45.91.116.12:4226 , piece_id=5QFSWOPQLZO2KEP4J27RDULUD4VCQSIE6BZ6BO3MFVFAP67OMW7A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.047677267 (11h33m18.637022254s) [noise=true , address=104.36.82.15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082384121 (11h33m18.671729108s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082409660 (11h33m18.671754647s) [seg_pos={0 0}, upload=12046]: Segment reached optimal threshold of 13 pieces. Aug 7 21:13:19.082628518 (11h33m18.671973495s) [noise=true , address=77.90.31.14:4661 , piece_id=P4DQFG2ACQKL52JQZFCTIFCUHFPLDIYEOOG4OPBJY4MIXBVKCYDQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Aug 7 21:13:19.082795017 (11h33m18.672139994s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: read tcp 188.166.225.170:56758->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082822870 (11h33m18.672167857s) [noise=true , address=104.36.82.16:4294 , piece_id=QN2G4ZPHQ2VBVMEJFPY3OLOGLHZRKEFC4SXOXSHFBQMD47QMAM7Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:47752->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082839501 (11h33m18.672184478s) [seg_pos={0 0}, upload=12046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:13:19.082910688 (11h33m18.672255675s) [upload=12046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:19.082914756 (11h33m18.672259743s) [upload=12046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:19.387715025 (11h33m18.977060012s) [upload=12046]: Done uploading segments... Aug 7 21:13:19.387739612 (11h33m18.977084599s) [upload=12046]: Upload finished. err: <nil> Aug 7 21:13:45.069592425 (11h33m44.658937412s) [upload=12058]: Starting upload Aug 7 21:13:45.069646459 (11h33m44.658991446s) [upload=12058]: Done starting upload Aug 7 21:13:45.069729608 (11h33m44.659074595s) [upload=12058]: Uploading segments... Aug 7 21:13:45.069894263 (11h33m44.65923925s) [upload=12058]: Got next segment. Inline: false Aug 7 21:13:45.069927166 (11h33m44.659272153s) [upload=12058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.069931384 (11h33m44.659276371s) [upload=12058]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:14.428211569 (11h34m14.017556556s) [upload=12058]: Next segment err: context deadline exceeded Aug 7 21:14:14.428287775 (11h34m14.017632762s) [upload=12058]: Done uploading segments... Aug 7 21:14:14.428338853 (11h34m14.01768384s) [upload=12058]: Upload finished. err: context deadline exceeded Aug 7 21:14:49.077304183 (11h34m48.66664917s) [upload=12074]: Starting upload Aug 7 21:14:49.077411358 (11h34m48.666756365s) [upload=12074]: Done starting upload Aug 7 21:14:49.077536547 (11h34m48.666881534s) [upload=12074]: Uploading segments... Aug 7 21:14:49.077698226 (11h34m48.667043214s) [upload=12074]: Got next segment. Inline: false Aug 7 21:14:49.077743253 (11h34m48.66708824s) [upload=12074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:49.077748543 (11h34m48.66709353s) [upload=12074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:18.426190970 (11h35m18.015535997s) [upload=12074]: Next segment err: context deadline exceeded Aug 7 21:15:18.426350985 (11h35m18.015695972s) [upload=12074]: Done uploading segments... Aug 7 21:15:18.426379730 (11h35m18.015724707s) [upload=12074]: Upload finished. err: context deadline exceeded Duration: 30.000777412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......XX.......................X............................] duration: 1.947190072s FAIL (2025-08-07 21:15:16 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== pb.BatchRequestItem_ObjectCommit Aug 7 20:43:37.067799677 (11h3m36.657144664s) [upload=11534]: Done uploading segments... Aug 7 20:43:37.067817170 (11h3m36.657162157s) [upload=11534]: Upload finished. err: <nil> Aug 7 20:44:07.448276030 (11h4m7.037621017s) [upload=11539]: Starting upload Aug 7 20:44:07.448326927 (11h4m7.037671904s) [upload=11539]: Done starting upload Aug 7 20:44:07.448378606 (11h4m7.037723593s) [upload=11539]: Uploading segments... Aug 7 20:44:07.448408994 (11h4m7.037753981s) [upload=11539]: Got next segment. Inline: true Aug 7 20:44:07.448414504 (11h4m7.037759491s) [upload=11539]: Waiting for error group managing segments... Aug 7 20:44:07.448448068 (11h4m7.037793055s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:44:07.448451785 (11h4m7.037796762s) [upload=11539]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:44:07.448454070 (11h4m7.037799057s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:44:07.799942249 (11h4m7.389287236s) [upload=11539]: Done uploading segments... Aug 7 20:44:07.799959912 (11h4m7.389304899s) [upload=11539]: Upload finished. err: <nil> Aug 7 20:45:19.440040944 (11h5m19.029385921s) [upload=11564]: Starting upload Aug 7 20:45:19.440090409 (11h5m19.029435386s) [upload=11564]: Done starting upload Aug 7 20:45:19.440146045 (11h5m19.029491032s) [upload=11564]: Uploading segments... Aug 7 20:45:19.440190359 (11h5m19.029535346s) [upload=11564]: Got next segment. Inline: true Aug 7 20:45:19.440194267 (11h5m19.029539254s) [upload=11564]: Waiting for error group managing segments... Aug 7 20:45:19.440226098 (11h5m19.029571085s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:45:19.440230026 (11h5m19.029575013s) [upload=11564]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:45:19.440232290 (11h5m19.029577277s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:45:19.858443364 (11h5m19.447788352s) [upload=11564]: Done uploading segments... Aug 7 20:45:19.858467080 (11h5m19.447812067s) [upload=11564]: Upload finished. err: <nil> Aug 7 20:46:00.840349205 (11h6m0.429694192s) [upload=11575]: Starting upload Aug 7 20:46:00.840387950 (11h6m0.429732927s) [upload=11575]: Done starting upload Aug 7 20:46:00.840431102 (11h6m0.429776079s) [upload=11575]: Uploading segments... Aug 7 20:46:00.840473594 (11h6m0.429818581s) [upload=11575]: Got next segment. Inline: true Aug 7 20:46:00.840477311 (11h6m0.429822298s) [upload=11575]: Waiting for error group managing segments... Aug 7 20:46:00.840510584 (11h6m0.429855571s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:00.840514551 (11h6m0.429859539s) [upload=11575]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:00.840516746 (11h6m0.429861733s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:01.149526394 (11h6m0.738871381s) [upload=11575]: Done uploading segments... Aug 7 20:46:01.149544077 (11h6m0.738889064s) [upload=11575]: Upload finished. err: <nil> Aug 7 20:46:32.531571609 (11h6m32.120916586s) [upload=11585]: Starting upload Aug 7 20:46:32.531627656 (11h6m32.120972643s) [upload=11585]: Done starting upload Aug 7 20:46:32.531679334 (11h6m32.121024311s) [upload=11585]: Uploading segments... Aug 7 20:46:32.531716416 (11h6m32.121061403s) [upload=11585]: Got next segment. Inline: true Aug 7 20:46:32.531720834 (11h6m32.121065821s) [upload=11585]: Waiting for error group managing segments... Aug 7 20:46:32.531758977 (11h6m32.121103964s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:32.531762984 (11h6m32.121107961s) [upload=11585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:32.531766161 (11h6m32.121111148s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:32.823368074 (11h6m32.412713051s) [upload=11585]: Done uploading segments... Aug 7 20:46:32.823385747 (11h6m32.412730724s) [upload=11585]: Upload finished. err: <nil> Aug 7 20:46:53.991475826 (11h6m53.580820813s) [upload=11590]: Starting upload Aug 7 20:46:53.991528908 (11h6m53.580873895s) [upload=11590]: Done starting upload Aug 7 20:46:53.991580286 (11h6m53.580925273s) [upload=11590]: Uploading segments... Aug 7 20:46:53.991609822 (11h6m53.58095481s) [upload=11590]: Got next segment. Inline: true Aug 7 20:46:53.991613780 (11h6m53.580958767s) [upload=11590]: Waiting for error group managing segments... Aug 7 20:46:53.991645681 (11h6m53.580990668s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:53.991649578 (11h6m53.580994565s) [upload=11590]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:53.991651822 (11h6m53.580996809s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:54.273031788 (11h6m53.862376835s) [upload=11590]: Done uploading segments... Aug 7 20:46:54.273056015 (11h6m53.862401032s) [upload=11590]: Upload finished. err: <nil> Aug 7 20:48:05.936480351 (11h8m5.525825338s) [upload=11610]: Starting upload Aug 7 20:48:05.936525999 (11h8m5.525870986s) [upload=11610]: Done starting upload Aug 7 20:48:05.936586324 (11h8m5.525931311s) [upload=11610]: Uploading segments... Aug 7 20:48:05.936615510 (11h8m5.525960487s) [upload=11610]: Got next segment. Inline: true Aug 7 20:48:05.936618666 (11h8m5.525963653s) [upload=11610]: Waiting for error group managing segments... Aug 7 20:48:05.936688099 (11h8m5.526033086s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:48:05.936692588 (11h8m5.526037575s) [upload=11610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:48:05.936694892 (11h8m5.526039879s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:48:06.223792648 (11h8m5.813137635s) [upload=11610]: Done uploading segments... Aug 7 20:48:06.223807756 (11h8m5.813152743s) [upload=11610]: Upload finished. err: <nil> Aug 7 20:49:00.370365369 (11h8m59.959710356s) [upload=11624]: Starting upload Aug 7 20:49:00.370414633 (11h8m59.95975962s) [upload=11624]: Done starting upload Aug 7 20:49:00.370467765 (11h8m59.959812752s) [upload=11624]: Uploading segments... Aug 7 20:49:00.370501249 (11h8m59.959846236s) [upload=11624]: Got next segment. Inline: true Aug 7 20:49:00.370504986 (11h8m59.959849973s) [upload=11624]: Waiting for error group managing segments... Aug 7 20:49:00.370538900 (11h8m59.959883887s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:00.370542898 (11h8m59.959887885s) [upload=11624]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:00.370545453 (11h8m59.95989043s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:00.736326238 (11h9m0.325671225s) [upload=11624]: Done uploading segments... Aug 7 20:49:00.736343821 (11h9m0.325688808s) [upload=11624]: Upload finished. err: <nil> Aug 7 20:49:43.904464651 (11h9m43.493809639s) [upload=11637]: Starting upload Aug 7 20:49:43.904540737 (11h9m43.493885724s) [upload=11637]: Done starting upload Aug 7 20:49:43.904599109 (11h9m43.493944096s) [upload=11637]: Uploading segments... Aug 7 20:49:43.904635608 (11h9m43.493980595s) [upload=11637]: Got next segment. Inline: true Aug 7 20:49:43.904639476 (11h9m43.493984463s) [upload=11637]: Waiting for error group managing segments... Aug 7 20:49:43.904675195 (11h9m43.494020182s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:43.904679252 (11h9m43.494024239s) [upload=11637]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:43.904683430 (11h9m43.494028417s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:44.269042214 (11h9m43.858387201s) [upload=11637]: Done uploading segments... Aug 7 20:49:44.269058595 (11h9m43.858403582s) [upload=11637]: Upload finished. err: <nil> Aug 7 20:50:51.186360319 (11h10m50.775705306s) [upload=11656]: Starting upload Aug 7 20:50:51.186464308 (11h10m50.775809295s) [upload=11656]: Done starting upload Aug 7 20:50:51.186527319 (11h10m50.775872306s) [upload=11656]: Uploading segments... Aug 7 20:50:51.186558759 (11h10m50.775903746s) [upload=11656]: Got next segment. Inline: true Aug 7 20:50:51.186562295 (11h10m50.775907282s) [upload=11656]: Waiting for error group managing segments... Aug 7 20:50:51.186595108 (11h10m50.775940095s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:50:51.186599105 (11h10m50.775944082s) [upload=11656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:50:51.186601591 (11h10m50.775946578s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:50:51.466357188 (11h10m51.055702175s) [upload=11656]: Done uploading segments... Aug 7 20:50:51.466375784 (11h10m51.055720781s) [upload=11656]: Upload finished. err: <nil> Aug 7 20:51:54.183767656 (11h11m53.773112643s) [upload=11674]: Starting upload Aug 7 20:51:54.183810869 (11h11m53.773155846s) [upload=11674]: Done starting upload Aug 7 20:51:54.183850504 (11h11m53.773195481s) [upload=11674]: Uploading segments... Aug 7 20:51:54.183877035 (11h11m53.773222012s) [upload=11674]: Got next segment. Inline: true Aug 7 20:51:54.183880662 (11h11m53.773225639s) [upload=11674]: Waiting for error group managing segments... Aug 7 20:51:54.183910529 (11h11m53.773255516s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:51:54.183914426 (11h11m53.773259413s) [upload=11674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:51:54.183916961 (11h11m53.773261938s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:51:54.662596438 (11h11m54.251941425s) [upload=11674]: Done uploading segments... Aug 7 20:51:54.662619702 (11h11m54.251964689s) [upload=11674]: Upload finished. err: <nil> Duration:* 1.94721589s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.00129253s FAIL (2025-08-07 21:15:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001306096s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 30.001260968s FAIL (2025-08-07 21:15:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00128197s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.000893097s FAIL (2025-08-07 21:15:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000906121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000745929s FAIL (2025-08-07 21:15:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [upload=11945]: Upload finished. err: <nil> Aug 7 21:08:20.470385011 (11h28m20.059729988s) [upload=11961]: Starting upload Aug 7 21:08:20.470444956 (11h28m20.059789943s) [upload=11961]: Done starting upload Aug 7 21:08:20.470517715 (11h28m20.059862702s) [upload=11961]: Uploading segments... Aug 7 21:08:20.470590324 (11h28m20.059935311s) [upload=11961]: Got next segment. Inline: true Aug 7 21:08:20.470594982 (11h28m20.059939971s) [upload=11961]: Waiting for error group managing segments... Aug 7 21:08:20.470639418 (11h28m20.059984405s) [upload=11961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:08:20.470645349 (11h28m20.059990336s) [upload=11961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:08:20.470647794 (11h28m20.059992781s) [upload=11961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:08:20.752099520 (11h28m20.341444507s) [upload=11961]: Done uploading segments... Aug 7 21:08:20.752117565 (11h28m20.341462552s) [upload=11961]: Upload finished. err: <nil> Aug 7 21:09:25.498062703 (11h29m25.08740769s) [upload=11978]: Starting upload Aug 7 21:09:25.498115193 (11h29m25.08746018s) [upload=11978]: Done starting upload Aug 7 21:09:25.498162373 (11h29m25.08750736s) [upload=11978]: Uploading segments... Aug 7 21:09:25.498190667 (11h29m25.087535654s) [upload=11978]: Got next segment. Inline: true Aug 7 21:09:25.498196007 (11h29m25.087540994s) [upload=11978]: Waiting for error group managing segments... Aug 7 21:09:25.498229983 (11h29m25.08757497s) [upload=11978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:25.498233650 (11h29m25.087578637s) [upload=11978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:25.498236134 (11h29m25.087581121s) [upload=11978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:25.808420372 (11h29m25.397765359s) [upload=11978]: Done uploading segments... Aug 7 21:09:25.808436974 (11h29m25.397781961s) [upload=11978]: Upload finished. err: <nil> Aug 7 21:09:47.139455442 (11h29m46.728800429s) [upload=11988]: Starting upload Aug 7 21:09:47.139503314 (11h29m46.728848301s) [upload=11988]: Done starting upload Aug 7 21:09:47.139548831 (11h29m46.728893818s) [upload=11988]: Uploading segments... Aug 7 21:09:47.139579450 (11h29m46.728924437s) [upload=11988]: Got next segment. Inline: true Aug 7 21:09:47.139583277 (11h29m46.728928264s) [upload=11988]: Waiting for error group managing segments... Aug 7 21:09:47.139618274 (11h29m46.728963261s) [upload=11988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:47.139622451 (11h29m46.728967439s) [upload=11988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:47.139624797 (11h29m46.728969784s) [upload=11988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.434297548 (11h29m47.023642535s) [upload=11988]: Done uploading segments... Aug 7 21:09:47.434326894 (11h29m47.023671872s) [upload=11988]: Upload finished. err: <nil> Aug 7 21:10:17.433462616 (11h30m17.022807593s) [upload=11995]: Starting upload Aug 7 21:10:17.433520777 (11h30m17.022865764s) [upload=11995]: Done starting upload Aug 7 21:10:17.433584859 (11h30m17.022929846s) [upload=11995]: Uploading segments... Aug 7 21:10:17.433692035 (11h30m17.023037012s) [upload=11995]: Got next segment. Inline: true Aug 7 21:10:17.433696885 (11h30m17.023041872s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:10:17.433744876 (11h30m17.023089853s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:17.433749755 (11h30m17.023094742s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:17.433754204 (11h30m17.023099191s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:17.710858662 (11h30m17.300203709s) [upload=11995]: Done uploading segments... Aug 7 21:10:17.710881997 (11h30m17.300227024s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:10:36.218517401 (11h30m35.807862378s) [upload=11998]: Starting upload Aug 7 21:10:36.218649954 (11h30m35.807994941s) [upload=11998]: Done starting upload Aug 7 21:10:36.218713896 (11h30m35.808058913s) [upload=11998]: Uploading segments... Aug 7 21:10:36.218766106 (11h30m35.808111103s) [upload=11998]: Got next segment. Inline: true Aug 7 21:10:36.218772398 (11h30m35.808117425s) [upload=11998]: Waiting for error group managing segments... Aug 7 21:10:36.218820571 (11h30m35.808165558s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:36.218825550 (11h30m35.808170537s) [upload=11998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:36.218828946 (11h30m35.808173943s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:36.494741534 (11h30m36.084086521s) [upload=11998]: Done uploading segments... Aug 7 21:10:36.494760030 (11h30m36.084105007s) [upload=11998]: Upload finished. err: <nil> Aug 7 21:11:36.918559943 (11h31m36.50790493s) [upload=12018]: Starting upload Aug 7 21:11:36.918615830 (11h31m36.507960817s) [upload=12018]: Done starting upload Aug 7 21:11:36.918660295 (11h31m36.508005282s) [upload=12018]: Uploading segments... Aug 7 21:11:36.918688629 (11h31m36.508033606s) [upload=12018]: Got next segment. Inline: true Aug 7 21:11:36.918691745 (11h31m36.508036732s) [upload=12018]: Waiting for error group managing segments... Aug 7 21:11:36.918721552 (11h31m36.508066539s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:36.918725530 (11h31m36.508070517s) [upload=12018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:36.918727794 (11h31m36.508072781s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:37.211063805 (11h31m36.800408792s) [upload=12018]: Done uploading segments... Aug 7 21:11:37.211087029 (11h31m36.800432026s) [upload=12018]: Upload finished. err: <nil> Aug 7 21:12:15.260577538 (11h32m14.849922525s) [upload=12032]: Starting upload Aug 7 21:12:15.260632023 (11h32m14.84997701s) [upload=12032]: Done starting upload Aug 7 21:12:15.260678571 (11h32m14.850023548s) [upload=12032]: Uploading segments... Aug 7 21:12:15.260706434 (11h32m14.850051421s) [upload=12032]: Got next segment. Inline: true Aug 7 21:12:15.260710011 (11h32m14.850054998s) [upload=12032]: Waiting for error group managing segments... Aug 7 21:12:15.260743897 (11h32m14.850088874s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:15.260747644 (11h32m14.850092631s) [upload=12032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:15.260750389 (11h32m14.850095376s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:15.565323071 (11h32m15.154668058s) [upload=12032]: Done uploading segments... Aug 7 21:12:15.565341296 (11h32m15.154686283s) [upload=12032]: Upload finished. err: <nil> Aug 7 21:12:41.001560374 (11h32m40.590905361s) [upload=12034]: Starting upload Aug 7 21:12:41.001659143 (11h32m40.59100413s) [upload=12034]: Done starting upload Aug 7 21:12:41.001745669 (11h32m40.591090666s) [upload=12034]: Uploading segments... Aug 7 21:12:41.001852704 (11h32m40.591197701s) [upload=12034]: Got next segment. Inline: true Aug 7 21:12:41.001857372 (11h32m40.591202359s) [upload=12034]: Waiting for error group managing segments... Aug 7 21:12:41.001907849 (11h32m40.591252836s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:41.001913169 (11h32m40.591258156s) [upload=12034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:41.001915584 (11h32m40.591260571s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:41.286244505 (11h32m40.875589493s) [upload=12034]: Done uploading segments... Aug 7 21:12:41.286260567 (11h32m40.875605544s) [upload=12034]: Upload finished. err: <nil> Aug 7 21:13:16.624584146 (11h33m16.213929123s) [upload=12047]: Starting upload Aug 7 21:13:16.624659630 (11h33m16.214004607s) [upload=12047]: Done starting upload Aug 7 21:13:16.624710127 (11h33m16.214055114s) [upload=12047]: Uploading segments... Aug 7 21:13:16.624745715 (11h33m16.214090702s) [upload=12047]: Got next segment. Inline: true Aug 7 21:13:16.624750654 (11h33m16.214095641s) [upload=12047]: Waiting for error group managing segments... Aug 7 21:13:16.624782977 (11h33m16.214127964s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:16.624786804 (11h33m16.214131791s) [upload=12047]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:16.624789339 (11h33m16.214134326s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.974391598 (11h33m45.563736596s) [upload=12047]: Done uploading segments... Aug 7 21:13:45.974419923 (11h33m45.56376491s) [upload=12047]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:31.328315035 (11h34m30.917660012s) [upload=12067]: Starting upload Aug 7 21:14:31.328377385 (11h34m30.917722373s) [upload=12067]: Done starting upload Aug 7 21:14:31.328450665 (11h34m30.917795652s) [upload=12067]: Uploading segments... Aug 7 21:14:31.328587126 (11h34m30.917932113s) [upload=12067]: Got next segment. Inline: true Aug 7 21:14:31.328592336 (11h34m30.917937323s) [upload=12067]: Waiting for error group managing segments... Aug 7 21:14:31.328654706 (11h34m30.917999693s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:31.328660026 (11h34m30.918005013s) [upload=12067]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:31.328662631 (11h34m30.918007618s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:00.687780769 (11h35m0.277125756s) [upload=12067]: Done uploading segments... Aug 7 21:15:00.687819252 (11h35m0.27716424s) [upload=12067]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.0007675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XX...........X.......................X...] duration: 5.899012358s FAIL (2025-08-07 21:14:55 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 7106407 (11h31m46.936451394s) [upload=12022]: Uploading segments... Aug 7 21:11:47.347198184 (11h31m46.936543161s) [upload=12022]: Got next segment. Inline: false Aug 7 21:11:47.347234122 (11h31m46.936579109s) [upload=12022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:47.347237969 (11h31m46.936582956s) [upload=12022]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:11:47.802049075 (11h31m47.391394053s) [seg_pos={0 0}, upload=12022]: Begin upload segment... Aug 7 21:11:47.802118007 (11h31m47.391462994s) [seg_pos={0 0}, upload=12022]: Done begin upload segment. Aug 7 21:11:47.802142414 (11h31m47.391487401s) [noise=true , address=104.36.82.22:4541 , piece_id=Q5X72MXPD2VX3TRLK37VB3OV5Y4WLD4P6CWPAVL2X4D6BSIIOFSQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802186428 (11h31m47.391531415s) [noise=true , address=104.36.82.5:4846 , piece_id=6NNDPTLCBYLGFWU2H57AIRLEEXTREPMET5R2YZETWQYPIP2ZUHHQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802550154 (11h31m47.391895141s) [noise=true , address=104.36.82.46:4501 , piece_id=CVPKLEXG5FMF3NAR3ONF7LGIWUXJ7OSF5BBOTJ3BLQE4CFFNQKBA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802702837 (11h31m47.392047814s) [noise=true , address=45.91.116.9:4268 , piece_id=VJP2ZZ3UMB2XMXGDGIV45YOLKHRNNJORX2NIB3SHBCI4UZNJEN7A, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802844608 (11h31m47.392189595s) [noise=true , address=104.36.82.16:4328 , piece_id=BFL3YNCSEKYNXJV6PJDPERHGUC6PDQZ52JNSLZDZPH3DCLYCAJPA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803081390 (11h31m47.392426377s) [noise=true , address=104.36.82.47:4523 , piece_id=4WH6OVD7YSUMLPUBHAQPDR5VYX7Y36WKS3MSM7ULY6CI73BALULQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803116328 (11h31m47.392461315s) [noise=true , address=45.91.116.11:4150 , piece_id=64KLQELYVF4SAJJYCFSQD3X5USFFYLXGNOJOQGZ36LOYOFEJ2UBQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803402735 (11h31m47.392747722s) [noise=true , address=104.36.82.12:4165 , piece_id=BBVGOIQK3IBDXSA5KXGHV4Z4TRPZYJQGJT3PYQJXZUI7ZJ55KRJA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803414909 (11h31m47.392759896s) [noise=true , address=104.36.82.39:4752 , piece_id=ITMK3WWPOHUBVDUV2ZUL65XLBGCIVZKGUD544R2K3FCAMTQN5TXQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803693041 (11h31m47.393038028s) [noise=true , address=104.36.82.36:4622 , piece_id=JYCNDCLZYJWGEB4ZWJM5ES3G25WPLBPOY6UGXI37SFC467KTIZEA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803929323 (11h31m47.39327431s) [noise=true , address=104.36.82.42:4813 , piece_id=UAXL4ZUQ4W6P3DLPKIWW7KC3U2MZPZQ25ZA7Z4A7BFQ42EMJZAXQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804163892 (11h31m47.393508879s) [noise=true , address=104.36.82.58:4269 , piece_id=QX6CM6DHSIW2KXY2Z6NRMSRG3HMBRNBQ3JRPHXBGOQMEL5HVZ5ZQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804383622 (11h31m47.393728609s) [noise=true , address=45.91.116.4:4969 , piece_id=GVQDU7BRCLA6PHFSRGOJZBAV6QZQRI2ZEG3EM6I2TJAZ4JAE4P7Q, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804621557 (11h31m47.393966544s) [noise=true , address=104.36.82.23:4453 , piece_id=BLRIGJ5Z3NNUPUHC2YZW4PDMK2SDUVPE75GO5T4WSFTT2FDOEEOQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804837270 (11h31m47.394182257s) [noise=true , address=104.36.82.2:4744 , piece_id=V6TRPHHDLGWKVLQH7BT6AVYVOKVVP3EHL67D62SPHAKNMWICYJWQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804911372 (11h31m47.394256359s) [noise=true , address=104.36.82.43:4849 , piece_id=IZH45DQEPVT7WEQ2UDAAA7OS37XKMKG4VBJRRUEKQ3G36YUM2LMQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.805071538 (11h31m47.394416515s) [noise=true , address=64.49.13.170:4728 , piece_id=LED5VGYENQQBGOP3UH4NKT6O6TH4M6OEZCIXZ5N7AYWFLW452KSA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.805261793 (11h31m47.39460677s) [noise=true , address=104.36.82.44:4899 , piece_id=I5PVQ6WHWORI2TFF2SJSZDAUDH3FX5DVNPP23VC6JECMDAO6ZM6Q, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:48.969580099 (11h31m48.558925086s) [upload=12022]: Next returned nil segment Aug 7 21:11:48.969596991 (11h31m48.558941978s) [upload=12022]: Waiting for error group managing segments... Aug 7 21:11:49.265222255 (11h31m48.854567242s) [noise=true , address=64.49.13.170:4728 , piece_id=LED5VGYENQQBGOP3UH4NKT6O6TH4M6OEZCIXZ5N7AYWFLW452KSA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.737280759 (11h31m49.326625746s) [noise=true , address=104.36.82.58:4269 , piece_id=QX6CM6DHSIW2KXY2Z6NRMSRG3HMBRNBQ3JRPHXBGOQMEL5HVZ5ZQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.753249862 (11h31m49.342594849s) [noise=true , address=104.36.82.12:4165 , piece_id=BBVGOIQK3IBDXSA5KXGHV4Z4TRPZYJQGJT3PYQJXZUI7ZJ55KRJA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.780910703 (11h31m49.37025569s) [noise=true , address=45.91.116.9:4268 , piece_id=VJP2ZZ3UMB2XMXGDGIV45YOLKHRNNJORX2NIB3SHBCI4UZNJEN7A, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.813098015 (11h31m49.402442992s) [noise=true , address=104.36.82.46:4501 , piece_id=CVPKLEXG5FMF3NAR3ONF7LGIWUXJ7OSF5BBOTJ3BLQE4CFFNQKBA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.827616893 (11h31m49.41696188s) [noise=true , address=104.36.82.42:4813 , piece_id=UAXL4ZUQ4W6P3DLPKIWW7KC3U2MZPZQ25ZA7Z4A7BFQ42EMJZAXQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.844644011 (11h31m49.433988998s) [noise=true , address=104.36.82.5:4846 , piece_id=6NNDPTLCBYLGFWU2H57AIRLEEXTREPMET5R2YZETWQYPIP2ZUHHQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.896979020 (11h31m49.486324007s) [noise=true , address=104.36.82.16:4328 , piece_id=BFL3YNCSEKYNXJV6PJDPERHGUC6PDQZ52JNSLZDZPH3DCLYCAJPA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.913302922 (11h31m49.502647909s) [noise=true , address=104.36.82.36:4622 , piece_id=JYCNDCLZYJWGEB4ZWJM5ES3G25WPLBPOY6UGXI37SFC467KTIZEA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.928574831 (11h31m49.517919809s) [noise=true , address=104.36.82.22:4541 , piece_id=Q5X72MXPD2VX3TRLK37VB3OV5Y4WLD4P6CWPAVL2X4D6BSIIOFSQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.967608171 (11h31m49.556953148s) [noise=true , address=104.36.82.43:4849 , piece_id=IZH45DQEPVT7WEQ2UDAAA7OS37XKMKG4VBJRRUEKQ3G36YUM2LMQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.087000568 (11h31m49.676345555s) [noise=true , address=45.91.116.11:4150 , piece_id=64KLQELYVF4SAJJYCFSQD3X5USFFYLXGNOJOQGZ36LOYOFEJ2UBQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.098037264 (11h31m49.687382241s) [noise=true , address=45.91.116.4:4969 , piece_id=GVQDU7BRCLA6PHFSRGOJZBAV6QZQRI2ZEG3EM6I2TJAZ4JAE4P7Q, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.160148408 (11h31m49.749493395s) [noise=true , address=104.36.82.47:4523 , piece_id=4WH6OVD7YSUMLPUBHAQPDR5VYX7Y36WKS3MSM7ULY6CI73BALULQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.160172064 (11h31m49.749517051s) [seg_pos={0 0}, upload=12022]: Segment reached optimal threshold of 14 pieces. Aug 7 21:11:50.160367668 (11h31m49.749712655s) [noise=true , address=104.36.82.44:4899 , piece_id=I5PVQ6WHWORI2TFF2SJSZDAUDH3FX5DVNPP23VC6JECMDAO6ZM6Q, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.160428794 (11h31m49.749773782s) [noise=true , address=104.36.82.23:4453 , piece_id=BLRIGJ5Z3NNUPUHC2YZW4PDMK2SDUVPE75GO5T4WSFTT2FDOEEOQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.160601615 (11h31m49.749946592s) [noise=true , address=104.36.82.2:4744 , piece_id=V6TRPHHDLGWKVLQH7BT6AVYVOKVVP3EHL67D62SPHAKNMWICYJWQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: read tcp 188.166.225.170:55148->104.36.82.2:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:11:50.160658343 (11h31m49.75000333s) [noise=true , address=104.36.82.39:4752 , piece_id=ITMK3WWPOHUBVDUV2ZUL65XLBGCIVZKGUD544R2K3FCAMTQN5TXQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: read tcp 188.166.225.170:46618->104.36.82.39:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:11:50.160688220 (11h31m49.750033207s) [seg_pos={0 0}, upload=12022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:11:50.160746132 (11h31m49.750091179s) [upload=12022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:11:50.160752644 (11h31m49.750097651s) [upload=12022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:50.684435488 (11h31m50.273780475s) [upload=12022]: Done uploading segments... Aug 7 21:11:50.684472599 (11h31m50.273817576s) [upload=12022]: Upload finished. err: <nil> Duration: 5.899041665s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000959303s FAIL (2025-08-07 21:14:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000978931s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......XX.......................X...........................] duration: 1.908399106s FAIL (2025-08-07 21:14:46 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== pb.BatchRequestItem_ObjectCommit Aug 7 20:43:37.067799677 (11h3m36.657144664s) [upload=11534]: Done uploading segments... Aug 7 20:43:37.067817170 (11h3m36.657162157s) [upload=11534]: Upload finished. err: <nil> Aug 7 20:44:07.448276030 (11h4m7.037621017s) [upload=11539]: Starting upload Aug 7 20:44:07.448326927 (11h4m7.037671904s) [upload=11539]: Done starting upload Aug 7 20:44:07.448378606 (11h4m7.037723593s) [upload=11539]: Uploading segments... Aug 7 20:44:07.448408994 (11h4m7.037753981s) [upload=11539]: Got next segment. Inline: true Aug 7 20:44:07.448414504 (11h4m7.037759491s) [upload=11539]: Waiting for error group managing segments... Aug 7 20:44:07.448448068 (11h4m7.037793055s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:44:07.448451785 (11h4m7.037796762s) [upload=11539]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:44:07.448454070 (11h4m7.037799057s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:44:07.799942249 (11h4m7.389287236s) [upload=11539]: Done uploading segments... Aug 7 20:44:07.799959912 (11h4m7.389304899s) [upload=11539]: Upload finished. err: <nil> Aug 7 20:45:19.440040944 (11h5m19.029385921s) [upload=11564]: Starting upload Aug 7 20:45:19.440090409 (11h5m19.029435386s) [upload=11564]: Done starting upload Aug 7 20:45:19.440146045 (11h5m19.029491032s) [upload=11564]: Uploading segments... Aug 7 20:45:19.440190359 (11h5m19.029535346s) [upload=11564]: Got next segment. Inline: true Aug 7 20:45:19.440194267 (11h5m19.029539254s) [upload=11564]: Waiting for error group managing segments... Aug 7 20:45:19.440226098 (11h5m19.029571085s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:45:19.440230026 (11h5m19.029575013s) [upload=11564]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:45:19.440232290 (11h5m19.029577277s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:45:19.858443364 (11h5m19.447788352s) [upload=11564]: Done uploading segments... Aug 7 20:45:19.858467080 (11h5m19.447812067s) [upload=11564]: Upload finished. err: <nil> Aug 7 20:46:00.840349205 (11h6m0.429694192s) [upload=11575]: Starting upload Aug 7 20:46:00.840387950 (11h6m0.429732927s) [upload=11575]: Done starting upload Aug 7 20:46:00.840431102 (11h6m0.429776079s) [upload=11575]: Uploading segments... Aug 7 20:46:00.840473594 (11h6m0.429818581s) [upload=11575]: Got next segment. Inline: true Aug 7 20:46:00.840477311 (11h6m0.429822298s) [upload=11575]: Waiting for error group managing segments... Aug 7 20:46:00.840510584 (11h6m0.429855571s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:00.840514551 (11h6m0.429859539s) [upload=11575]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:00.840516746 (11h6m0.429861733s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:01.149526394 (11h6m0.738871381s) [upload=11575]: Done uploading segments... Aug 7 20:46:01.149544077 (11h6m0.738889064s) [upload=11575]: Upload finished. err: <nil> Aug 7 20:46:32.531571609 (11h6m32.120916586s) [upload=11585]: Starting upload Aug 7 20:46:32.531627656 (11h6m32.120972643s) [upload=11585]: Done starting upload Aug 7 20:46:32.531679334 (11h6m32.121024311s) [upload=11585]: Uploading segments... Aug 7 20:46:32.531716416 (11h6m32.121061403s) [upload=11585]: Got next segment. Inline: true Aug 7 20:46:32.531720834 (11h6m32.121065821s) [upload=11585]: Waiting for error group managing segments... Aug 7 20:46:32.531758977 (11h6m32.121103964s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:32.531762984 (11h6m32.121107961s) [upload=11585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:32.531766161 (11h6m32.121111148s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:32.823368074 (11h6m32.412713051s) [upload=11585]: Done uploading segments... Aug 7 20:46:32.823385747 (11h6m32.412730724s) [upload=11585]: Upload finished. err: <nil> Aug 7 20:46:53.991475826 (11h6m53.580820813s) [upload=11590]: Starting upload Aug 7 20:46:53.991528908 (11h6m53.580873895s) [upload=11590]: Done starting upload Aug 7 20:46:53.991580286 (11h6m53.580925273s) [upload=11590]: Uploading segments... Aug 7 20:46:53.991609822 (11h6m53.58095481s) [upload=11590]: Got next segment. Inline: true Aug 7 20:46:53.991613780 (11h6m53.580958767s) [upload=11590]: Waiting for error group managing segments... Aug 7 20:46:53.991645681 (11h6m53.580990668s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:53.991649578 (11h6m53.580994565s) [upload=11590]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:53.991651822 (11h6m53.580996809s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:54.273031788 (11h6m53.862376835s) [upload=11590]: Done uploading segments... Aug 7 20:46:54.273056015 (11h6m53.862401032s) [upload=11590]: Upload finished. err: <nil> Aug 7 20:48:05.936480351 (11h8m5.525825338s) [upload=11610]: Starting upload Aug 7 20:48:05.936525999 (11h8m5.525870986s) [upload=11610]: Done starting upload Aug 7 20:48:05.936586324 (11h8m5.525931311s) [upload=11610]: Uploading segments... Aug 7 20:48:05.936615510 (11h8m5.525960487s) [upload=11610]: Got next segment. Inline: true Aug 7 20:48:05.936618666 (11h8m5.525963653s) [upload=11610]: Waiting for error group managing segments... Aug 7 20:48:05.936688099 (11h8m5.526033086s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:48:05.936692588 (11h8m5.526037575s) [upload=11610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:48:05.936694892 (11h8m5.526039879s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:48:06.223792648 (11h8m5.813137635s) [upload=11610]: Done uploading segments... Aug 7 20:48:06.223807756 (11h8m5.813152743s) [upload=11610]: Upload finished. err: <nil> Aug 7 20:49:00.370365369 (11h8m59.959710356s) [upload=11624]: Starting upload Aug 7 20:49:00.370414633 (11h8m59.95975962s) [upload=11624]: Done starting upload Aug 7 20:49:00.370467765 (11h8m59.959812752s) [upload=11624]: Uploading segments... Aug 7 20:49:00.370501249 (11h8m59.959846236s) [upload=11624]: Got next segment. Inline: true Aug 7 20:49:00.370504986 (11h8m59.959849973s) [upload=11624]: Waiting for error group managing segments... Aug 7 20:49:00.370538900 (11h8m59.959883887s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:00.370542898 (11h8m59.959887885s) [upload=11624]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:00.370545453 (11h8m59.95989043s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:00.736326238 (11h9m0.325671225s) [upload=11624]: Done uploading segments... Aug 7 20:49:00.736343821 (11h9m0.325688808s) [upload=11624]: Upload finished. err: <nil> Aug 7 20:49:43.904464651 (11h9m43.493809639s) [upload=11637]: Starting upload Aug 7 20:49:43.904540737 (11h9m43.493885724s) [upload=11637]: Done starting upload Aug 7 20:49:43.904599109 (11h9m43.493944096s) [upload=11637]: Uploading segments... Aug 7 20:49:43.904635608 (11h9m43.493980595s) [upload=11637]: Got next segment. Inline: true Aug 7 20:49:43.904639476 (11h9m43.493984463s) [upload=11637]: Waiting for error group managing segments... Aug 7 20:49:43.904675195 (11h9m43.494020182s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:43.904679252 (11h9m43.494024239s) [upload=11637]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:43.904683430 (11h9m43.494028417s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:44.269042214 (11h9m43.858387201s) [upload=11637]: Done uploading segments... Aug 7 20:49:44.269058595 (11h9m43.858403582s) [upload=11637]: Upload finished. err: <nil> Aug 7 20:50:51.186360319 (11h10m50.775705306s) [upload=11656]: Starting upload Aug 7 20:50:51.186464308 (11h10m50.775809295s) [upload=11656]: Done starting upload Aug 7 20:50:51.186527319 (11h10m50.775872306s) [upload=11656]: Uploading segments... Aug 7 20:50:51.186558759 (11h10m50.775903746s) [upload=11656]: Got next segment. Inline: true Aug 7 20:50:51.186562295 (11h10m50.775907282s) [upload=11656]: Waiting for error group managing segments... Aug 7 20:50:51.186595108 (11h10m50.775940095s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:50:51.186599105 (11h10m50.775944082s) [upload=11656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:50:51.186601591 (11h10m50.775946578s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:50:51.466357188 (11h10m51.055702175s) [upload=11656]: Done uploading segments... Aug 7 20:50:51.466375784 (11h10m51.055720781s) [upload=11656]: Upload finished. err: <nil> Aug 7 20:51:54.183767656 (11h11m53.773112643s) [upload=11674]: Starting upload Aug 7 20:51:54.183810869 (11h11m53.773155846s) [upload=11674]: Done starting upload Aug 7 20:51:54.183850504 (11h11m53.773195481s) [upload=11674]: Uploading segments... Aug 7 20:51:54.183877035 (11h11m53.773222012s) [upload=11674]: Got next segment. Inline: true Aug 7 20:51:54.183880662 (11h11m53.773225639s) [upload=11674]: Waiting for error group managing segments... Aug 7 20:51:54.183910529 (11h11m53.773255516s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:51:54.183914426 (11h11m53.773259413s) [upload=11674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:51:54.183916961 (11h11m53.773261938s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:51:54.662596438 (11h11m54.251941425s) [upload=11674]: Done uploading segments... Aug 7 20:51:54.662619702 (11h11m54.251964689s) [upload=11674]: Upload finished. err: <nil> Duration:* 1.908425868s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001515297s FAIL (2025-08-07 21:14:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 4VIN3PBQSMVEEFAYA3DF43KAALY6G6HCFADLZWIQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.232998345 (11h32m51.822343332s) [noise=true , address=104.36.82.6:4890 , piece_id=DBUDIACR7PEFAFGSOBGEIMWWYDQ4QD4ZWFQ5MNIT2DRDSHWJUE7A, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233068528 (11h32m51.822413516s) [noise=true , address=104.36.82.23:4453 , piece_id=OIB2N3NXKLOBJS2UV353TMJMCMK3QM4UGLNKX7P6PUZDT2MRNILQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233313337 (11h32m51.822658324s) [noise=true , address=104.36.82.55:4195 , piece_id=CKLDMZFNPKVSRXP5A6Z7DIRQOTMLENLOL7XBDWJTMIRDNCFV3ZIA, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233334317 (11h32m51.822679304s) [noise=true , address=104.36.82.43:4867 , piece_id=ZZCKCBXHV7POWCGCQ7DAJ2QE5A4UYFEERTSCUGVWU7UC76BYZTYA, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233595037 (11h32m51.822940024s) [noise=true , address=104.36.82.43:4866 , piece_id=AO5FAMAICZ2WFNV3L75ULE6UDCKGENWEBO47HXEA5ZVZ6KPOBFXQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233603493 (11h32m51.82294848s) [noise=true , address=104.36.82.18:4378 , piece_id=ZJJ542SCBSUKTG7QB5PQFTDRPHMFHN32RY6LXNSDLU7X6SCEVBNQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233891473 (11h32m51.82323646s) [noise=true , address=104.36.82.56:4247 , piece_id=X7EBBRNI7ZUNCR6SYWNS3TTBBQ4Z3BZZYWZNBYFBYRWYTV4TU5VQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.233989401 (11h32m51.823334388s) [noise=true , address=45.91.116.8:4067 , piece_id=GYAFL34K7CTRS3SOTQZFERYUQGQKVOBWUTPJ3E5CFYBZ7ON5RQVA, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234150710 (11h32m51.823495697s) [noise=true , address=45.91.116.7:4034 , piece_id=RDFD2PLXXJ3GRAOR5ZLPKDLRK5IGJTZSBSNVG4A6XFZB75WPJ22Q, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234254518 (11h32m51.823599506s) [noise=true , address=104.36.82.44:4882 , piece_id=M7WROH24EKE2EMGFKDDDS6UOM3E5P4OXLT2QDIBXN2BCYFZCYUMA, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234402521 (11h32m51.823747508s) [noise=true , address=104.36.82.62:4427 , piece_id=QTWU24TIM66G6SQXGSU4H22JZTWDPYSZVIZV4FGRILO77SGECVNQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234520336 (11h32m51.823865313s) [noise=true , address=104.36.82.60:4348 , piece_id=KYGK6JIIPPPB3WXEJME7RHT4YU2K5YA6XO2UFS7RVRFC2MK5HL3Q, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234683298 (11h32m51.824028285s) [noise=true , address=104.36.82.41:4772 , piece_id=BGC5NOQVRQ4BQEUPDI5UDRVKUBSTZMOQKGKLTHLTBNWFTORUWZQQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234718776 (11h32m51.824063753s) [noise=true , address=104.36.82.34:4571 , piece_id=APKT66JAXIQNA5UXXM3FCX7JCN5ARKXPDUD6ZJ6WB46MMR2VF3UQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.234915192 (11h32m51.824260179s) [noise=true , address=45.91.116.5:4325 , piece_id=P5KGQGDJXUKBS7F7ZK6TIP52U7I2OI74XFJTQSSQRFA2AS4WQDVQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:52.235686748 (11h32m51.825031735s) [noise=true , address=104.36.82.59:4333 , piece_id=BR6IGKYFNUAH2ZWKQMUQHYGMOL27HFFGN6PEPJOP7WODCYOVUJEQ, seg_pos={0 0}, upload=12039]: Uploading piece... Aug 7 21:12:53.755589114 (11h32m53.344934101s) [upload=12039]: Next returned nil segment Aug 7 21:12:53.755603641 (11h32m53.344948629s) [upload=12039]: Waiting for error group managing segments... Aug 7 21:12:54.242895872 (11h32m53.832240859s) [noise=true , address=104.36.82.44:4882 , piece_id=M7WROH24EKE2EMGFKDDDS6UOM3E5P4OXLT2QDIBXN2BCYFZCYUMA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.247833398 (11h32m53.837178375s) [noise=true , address=104.36.82.23:4453 , piece_id=OIB2N3NXKLOBJS2UV353TMJMCMK3QM4UGLNKX7P6PUZDT2MRNILQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.248473462 (11h32m53.837818449s) [noise=true , address=104.36.82.16:4300 , piece_id=RVRXMZLRANZJMWSQAF7HTJUULYADEA2I6GPHDTFGKNS6FNWE36VA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.248750062 (11h32m53.83809505s) [noise=true , address=104.36.82.60:4348 , piece_id=KYGK6JIIPPPB3WXEJME7RHT4YU2K5YA6XO2UFS7RVRFC2MK5HL3Q, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.249467144 (11h32m53.838812131s) [noise=true , address=104.36.82.18:4378 , piece_id=ZJJ542SCBSUKTG7QB5PQFTDRPHMFHN32RY6LXNSDLU7X6SCEVBNQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.269168297 (11h32m53.858513304s) [noise=true , address=104.36.82.56:4247 , piece_id=X7EBBRNI7ZUNCR6SYWNS3TTBBQ4Z3BZZYWZNBYFBYRWYTV4TU5VQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.286754543 (11h32m53.87609953s) [noise=true , address=104.36.82.34:4571 , piece_id=APKT66JAXIQNA5UXXM3FCX7JCN5ARKXPDUD6ZJ6WB46MMR2VF3UQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.315127418 (11h32m53.904472395s) [noise=true , address=104.36.82.43:4866 , piece_id=AO5FAMAICZ2WFNV3L75ULE6UDCKGENWEBO47HXEA5ZVZ6KPOBFXQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.339819320 (11h32m53.929164307s) [noise=true , address=104.36.82.55:4195 , piece_id=CKLDMZFNPKVSRXP5A6Z7DIRQOTMLENLOL7XBDWJTMIRDNCFV3ZIA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.352457851 (11h32m53.941802858s) [noise=true , address=104.36.82.59:4333 , piece_id=BR6IGKYFNUAH2ZWKQMUQHYGMOL27HFFGN6PEPJOP7WODCYOVUJEQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.366096104 (11h32m53.955441091s) [noise=true , address=104.36.82.6:4890 , piece_id=DBUDIACR7PEFAFGSOBGEIMWWYDQ4QD4ZWFQ5MNIT2DRDSHWJUE7A, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.366979544 (11h32m53.956324531s) [noise=true , address=104.36.82.62:4427 , piece_id=QTWU24TIM66G6SQXGSU4H22JZTWDPYSZVIZV4FGRILO77SGECVNQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.381141350 (11h32m53.970486347s) [noise=true , address=104.36.82.41:4772 , piece_id=BGC5NOQVRQ4BQEUPDI5UDRVKUBSTZMOQKGKLTHLTBNWFTORUWZQQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.391573819 (11h32m53.980918806s) [noise=true , address=45.91.116.8:4067 , piece_id=GYAFL34K7CTRS3SOTQZFERYUQGQKVOBWUTPJ3E5CFYBZ7ON5RQVA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: <nil> Aug 7 21:12:54.391594779 (11h32m53.980939766s) [seg_pos={0 0}, upload=12039]: Segment reached optimal threshold of 14 pieces. Aug 7 21:12:54.391723245 (11h32m53.981068232s) [noise=true , address=45.91.116.7:4034 , piece_id=RDFD2PLXXJ3GRAOR5ZLPKDLRK5IGJTZSBSNVG4A6XFZB75WPJ22Q, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391809851 (11h32m53.981154838s) [noise=true , address=45.91.116.5:4325 , piece_id=P5KGQGDJXUKBS7F7ZK6TIP52U7I2OI74XFJTQSSQRFA2AS4WQDVQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391822284 (11h32m53.981167271s) [noise=true , address=104.36.82.45:4939 , piece_id=UIVN6HS5ZIB44VIN3PBQSMVEEFAYA3DF43KAALY6G6HCFADLZWIQ, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu): protocol: write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection; write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54040->104.36.82.45:4939: use of closed network connection Aug 7 21:12:54.391862171 (11h32m53.981207158s) [noise=true , address=104.36.82.43:4867 , piece_id=ZZCKCBXHV7POWCGCQ7DAJ2QE5A4UYFEERTSCUGVWU7UC76BYZTYA, seg_pos={0 0}, upload=12039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zENB6bYmb9hqtGTcJ1afKVbcNtbEsGo4dpsSEdh2feqHifmHQ): protocol: expected piece hash; context canceled; EOF Aug 7 21:12:54.391878141 (11h32m53.981223118s) [seg_pos={0 0}, upload=12039]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:12:54.391977522 (11h32m53.981322509s) [upload=12039]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:12:54.391981659 (11h32m53.981326646s) [upload=12039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.855789778 (11h32m54.445134765s) [upload=12039]: Done uploading segments... Aug 7 21:12:54.855808144 (11h32m54.445153121s) [upload=12039]: Upload finished. err: <nil> Aug 7 21:13:45.303666216 (11h33m44.893011253s) [upload=12059]: Starting upload Aug 7 21:13:45.303786125 (11h33m44.893131112s) [upload=12059]: Done starting upload Aug 7 21:13:45.303856099 (11h33m44.893201126s) [upload=12059]: Uploading segments... Aug 7 21:13:45.304260994 (11h33m44.893605981s) [upload=12059]: Got next segment. Inline: false Aug 7 21:13:45.304300049 (11h33m44.893645036s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.304305609 (11h33m44.893650596s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:44.664782364 (11h34m44.254127351s) [upload=12059]: Next segment err: context deadline exceeded Aug 7 21:14:44.664969041 (11h34m44.254314028s) [upload=12059]: Done uploading segments... Aug 7 21:14:44.665004269 (11h34m44.254349256s) [upload=12059]: Upload finished. err: context deadline exceeded Duration: 1m0.001534515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m1.627880388s FAIL (2025-08-07 21:14:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m1.627906767s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 1m0.001095173s FAIL (2025-08-07 21:14:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 304s) [upload=11916]: Upload finished. err: <nil> Aug 7 21:06:29.023021686 (11h26m28.612366673s) [upload=11927]: Starting upload Aug 7 21:06:29.023068907 (11h26m28.612413894s) [upload=11927]: Done starting upload Aug 7 21:06:29.023149912 (11h26m28.612494899s) [upload=11927]: Uploading segments... Aug 7 21:06:29.023185480 (11h26m28.612530457s) [upload=11927]: Got next segment. Inline: true Aug 7 21:06:29.023189928 (11h26m28.612534915s) [upload=11927]: Waiting for error group managing segments... Aug 7 21:06:29.023231568 (11h26m28.612576555s) [upload=11927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:06:29.023236387 (11h26m28.612581374s) [upload=11927]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:06:29.023239593 (11h26m28.61258458s) [upload=11927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:06:29.300861462 (11h26m28.890206449s) [upload=11927]: Done uploading segments... Aug 7 21:06:29.300881371 (11h26m28.890226358s) [upload=11927]: Upload finished. err: <nil> Aug 7 21:07:37.321661864 (11h27m36.911006841s) [upload=11949]: Starting upload Aug 7 21:07:37.321718112 (11h27m36.911063099s) [upload=11949]: Done starting upload Aug 7 21:07:37.321771434 (11h27m36.911116411s) [upload=11949]: Uploading segments... Aug 7 21:07:37.321872968 (11h27m36.911217945s) [upload=11949]: Got next segment. Inline: true Aug 7 21:07:37.321877877 (11h27m36.911222865s) [upload=11949]: Waiting for error group managing segments... Aug 7 21:07:37.321923275 (11h27m36.911268262s) [upload=11949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:07:37.321928314 (11h27m36.911273301s) [upload=11949]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:07:37.321930418 (11h27m36.911275405s) [upload=11949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:07:37.599927314 (11h27m37.189272301s) [upload=11949]: Done uploading segments... Aug 7 21:07:37.599948635 (11h27m37.189293672s) [upload=11949]: Upload finished. err: <nil> Aug 7 21:08:35.851762080 (11h28m35.441107057s) [upload=11967]: Starting upload Aug 7 21:08:35.851828216 (11h28m35.441173203s) [upload=11967]: Done starting upload Aug 7 21:08:35.851888952 (11h28m35.441233939s) [upload=11967]: Uploading segments... Aug 7 21:08:35.851977722 (11h28m35.441322709s) [upload=11967]: Got next segment. Inline: true Aug 7 21:08:35.851983424 (11h28m35.441328411s) [upload=11967]: Waiting for error group managing segments... Aug 7 21:08:35.852017519 (11h28m35.441362506s) [upload=11967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:08:35.852021105 (11h28m35.441366093s) [upload=11967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:08:35.852023460 (11h28m35.441368447s) [upload=11967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:08:36.115286628 (11h28m35.704631615s) [upload=11967]: Done uploading segments... Aug 7 21:08:36.115331112 (11h28m35.704676099s) [upload=11967]: Upload finished. err: <nil> Aug 7 21:09:20.788493181 (11h29m20.377838178s) [upload=11976]: Starting upload Aug 7 21:09:20.788595918 (11h29m20.377940965s) [upload=11976]: Done starting upload Aug 7 21:09:20.788684948 (11h29m20.378029975s) [upload=11976]: Uploading segments... Aug 7 21:09:20.788747948 (11h29m20.378092935s) [upload=11976]: Got next segment. Inline: true Aug 7 21:09:20.788756344 (11h29m20.378101361s) [upload=11976]: Waiting for error group managing segments... Aug 7 21:09:20.788807752 (11h29m20.378152739s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:20.788813985 (11h29m20.378158972s) [upload=11976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:20.788818584 (11h29m20.378163611s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:21.070630248 (11h29m20.659975235s) [upload=11976]: Done uploading segments... Aug 7 21:09:21.070652190 (11h29m20.659997247s) [upload=11976]: Upload finished. err: <nil> Aug 7 21:09:46.948530536 (11h29m46.537875523s) [upload=11987]: Starting upload Aug 7 21:09:46.948588607 (11h29m46.537933584s) [upload=11987]: Done starting upload Aug 7 21:09:46.948642600 (11h29m46.537987587s) [upload=11987]: Uploading segments... Aug 7 21:09:46.948702786 (11h29m46.538047773s) [upload=11987]: Got next segment. Inline: true Aug 7 21:09:46.948707776 (11h29m46.538052763s) [upload=11987]: Waiting for error group managing segments... Aug 7 21:09:46.948747241 (11h29m46.538092228s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:46.948752340 (11h29m46.538097327s) [upload=11987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:46.948754896 (11h29m46.538099883s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.214261274 (11h29m46.803606261s) [upload=11987]: Done uploading segments... Aug 7 21:09:47.214279799 (11h29m46.803624786s) [upload=11987]: Upload finished. err: <nil> Aug 7 21:10:37.572355395 (11h30m37.161700372s) [upload=11999]: Starting upload Aug 7 21:10:37.572410881 (11h30m37.161755868s) [upload=11999]: Done starting upload Aug 7 21:10:37.572467810 (11h30m37.161812797s) [upload=11999]: Uploading segments... Aug 7 21:10:37.572531982 (11h30m37.161876959s) [upload=11999]: Got next segment. Inline: true Aug 7 21:10:37.572536320 (11h30m37.161881307s) [upload=11999]: Waiting for error group managing segments... Aug 7 21:10:37.572570917 (11h30m37.161915904s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:37.572574955 (11h30m37.161919942s) [upload=11999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:37.572577520 (11h30m37.161922507s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:37.838997375 (11h30m37.428342362s) [upload=11999]: Done uploading segments... Aug 7 21:10:37.839014699 (11h30m37.428359686s) [upload=11999]: Upload finished. err: <nil> Aug 7 21:11:31.258043368 (11h31m30.847388355s) [upload=12015]: Starting upload Aug 7 21:11:31.258101790 (11h31m30.847446777s) [upload=12015]: Done starting upload Aug 7 21:11:31.258163838 (11h31m30.847508825s) [upload=12015]: Uploading segments... Aug 7 21:11:31.258195730 (11h31m30.847540717s) [upload=12015]: Got next segment. Inline: true Aug 7 21:11:31.258199567 (11h31m30.847544565s) [upload=12015]: Waiting for error group managing segments... Aug 7 21:11:31.258240175 (11h31m30.847585162s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.258244853 (11h31m30.847589841s) [upload=12015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.258247358 (11h31m30.847592345s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.535852595 (11h31m31.125197582s) [upload=12015]: Done uploading segments... Aug 7 21:11:31.535871041 (11h31m31.125216028s) [upload=12015]: Upload finished. err: <nil> Aug 7 21:11:57.912240700 (11h31m57.501585687s) [upload=12025]: Starting upload Aug 7 21:11:57.912288532 (11h31m57.501633519s) [upload=12025]: Done starting upload Aug 7 21:11:57.912345871 (11h31m57.501690858s) [upload=12025]: Uploading segments... Aug 7 21:11:57.912374877 (11h31m57.501719874s) [upload=12025]: Got next segment. Inline: true Aug 7 21:11:57.912378794 (11h31m57.501723781s) [upload=12025]: Waiting for error group managing segments... Aug 7 21:11:57.912425353 (11h31m57.501770401s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:57.912431234 (11h31m57.501776232s) [upload=12025]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:57.912435933 (11h31m57.50178095s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:57.261749711 (11h32m56.851094698s) [upload=12025]: Done uploading segments... Aug 7 21:12:57.261780350 (11h32m56.851125337s) [upload=12025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:07.178414027 (11h33m6.767759014s) [upload=12044]: Starting upload Aug 7 21:13:07.178467459 (11h33m6.767812446s) [upload=12044]: Done starting upload Aug 7 21:13:07.178514029 (11h33m6.767859016s) [upload=12044]: Uploading segments... Aug 7 21:13:07.178543715 (11h33m6.767888703s) [upload=12044]: Got next segment. Inline: true Aug 7 21:13:07.178547282 (11h33m6.767892259s) [upload=12044]: Waiting for error group managing segments... Aug 7 21:13:07.178580015 (11h33m6.767925002s) [upload=12044]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:07.178583561 (11h33m6.767928548s) [upload=12044]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:07.178585735 (11h33m6.767930722s) [upload=12044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:07.471379642 (11h33m7.060724629s) [upload=12044]: Done uploading segments... Aug 7 21:13:07.471396245 (11h33m7.060741232s) [upload=12044]: Upload finished. err: <nil> Aug 7 21:13:32.020845488 (11h33m31.610190465s) [upload=12052]: Starting upload Aug 7 21:13:32.020909792 (11h33m31.610254769s) [upload=12052]: Done starting upload Aug 7 21:13:32.020999433 (11h33m31.61034443s) [upload=12052]: Uploading segments... Aug 7 21:13:32.021133279 (11h33m31.610478267s) [upload=12052]: Got next segment. Inline: true Aug 7 21:13:32.021141214 (11h33m31.610486231s) [upload=12052]: Waiting for error group managing segments... Aug 7 21:13:32.021222079 (11h33m31.610567066s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:32.021229403 (11h33m31.61057439s) [upload=12052]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:32.021234362 (11h33m31.610579399s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:31.378507697 (11h34m30.967852684s) [upload=12052]: Done uploading segments... Aug 7 21:14:31.378529929 (11h34m30.967874916s) [upload=12052]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001107497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 30.00097145s FAIL (2025-08-07 21:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== =12046]: Starting upload Aug 7 21:13:16.327545919 (11h33m15.916890906s) [upload=12046]: Done starting upload Aug 7 21:13:16.327700746 (11h33m15.917045733s) [upload=12046]: Uploading segments... Aug 7 21:13:16.327876632 (11h33m15.917221619s) [upload=12046]: Got next segment. Inline: false Aug 7 21:13:16.327968077 (11h33m15.917313064s) [upload=12046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:16.327974870 (11h33m15.917319857s) [upload=12046]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:13:16.564487897 (11h33m16.153832884s) [seg_pos={0 0}, upload=12046]: Begin upload segment... Aug 7 21:13:16.564542081 (11h33m16.153887068s) [seg_pos={0 0}, upload=12046]: Done begin upload segment. Aug 7 21:13:16.564705423 (11h33m16.1540504s) [noise=true , address=77.90.31.14:4661 , piece_id=P4DQFG2ACQKL52JQZFCTIFCUHFPLDIYEOOG4OPBJY4MIXBVKCYDQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.565371317 (11h33m16.154716304s) [noise=true , address=104.36.82.6:4873 , piece_id=VLAKUO36OHVGPY3Q723O5OUKURJ2SWREZC3SABUX2J7O6MBM7Y2Q, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.565974351 (11h33m16.155319338s) [noise=true , address=45.91.116.5:4299 , piece_id=5YQ5OJAPBA3N2KMYSVWQCNOTH7QPX2WCHHUXTLNSVNH2P7MZEJ3A, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.566321225 (11h33m16.155666212s) [noise=true , address=104.36.82.16:4294 , piece_id=QN2G4ZPHQ2VBVMEJFPY3OLOGLHZRKEFC4SXOXSHFBQMD47QMAM7Q, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.566822494 (11h33m16.156167471s) [noise=true , address=77.90.31.13:4626 , piece_id=S4OKGN6SKZ7GLYKCT3S4QSQFK75FX3KAXGQAOORJRVDJ7HV5UTWQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.567253970 (11h33m16.156598957s) [noise=true , address=45.91.116.5:4298 , piece_id=Y4FCHPKGJXDGWLH2FT6M5PJV5RA4UOYFQ3SHTWFJN4WO3FTFFG2A, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.568348233 (11h33m16.15769322s) [noise=true , address=45.91.116.12:4226 , piece_id=5QFSWOPQLZO2KEP4J27RDULUD4VCQSIE6BZ6BO3MFVFAP67OMW7A, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.570326768 (11h33m16.159671755s) [noise=true , address=77.90.31.11:4553 , piece_id=4QVTWAAY3JK3EQJIHK7ANW6E4UYUI3KXCHZOWRK75JHZZRPZ6ENQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.571062165 (11h33m16.160407143s) [noise=true , address=45.91.116.11:4119 , piece_id=6PDH442NIXRNQCIFX2Y4PNKU5C73T2H6KNHKGMIW2UGCTC5NX7ZA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.572859514 (11h33m16.162204501s) [noise=true , address=104.36.82.15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573225655 (11h33m16.162570642s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573527402 (11h33m16.162872389s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.573918310 (11h33m16.163263297s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574198737 (11h33m16.163543724s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574518488 (11h33m16.163863465s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:16.574831648 (11h33m16.164176635s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Uploading piece... Aug 7 21:13:18.308274980 (11h33m17.897619967s) [upload=12046]: Next returned nil segment Aug 7 21:13:18.308295720 (11h33m17.897640707s) [upload=12046]: Waiting for error group managing segments... Aug 7 21:13:18.656166252 (11h33m18.245511229s) [noise=true , address=77.90.31.13:4626 , piece_id=S4OKGN6SKZ7GLYKCT3S4QSQFK75FX3KAXGQAOORJRVDJ7HV5UTWQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.668187531 (11h33m18.257532518s) [noise=true , address=104.36.82.6:4873 , piece_id=VLAKUO36OHVGPY3Q723O5OUKURJ2SWREZC3SABUX2J7O6MBM7Y2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.721769396 (11h33m18.311114384s) [noise=true , address=104.36.82.12:4151 , piece_id=NTZGQ27R4C2QMPATFSFT5X6VSDWRHWQTC3B6SPWAONC2AZDOODXA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.741305974 (11h33m18.330650951s) [noise=true , address=77.90.31.21:4879 , piece_id=F4K4PUAMH6UE2WTGHQZ2OM2NPPAAYKJ3COWXQ5HQMJR4TP5FAH2Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.780529499 (11h33m18.369874486s) [noise=true , address=77.90.31.11:4553 , piece_id=4QVTWAAY3JK3EQJIHK7ANW6E4UYUI3KXCHZOWRK75JHZZRPZ6ENQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.865633524 (11h33m18.454978511s) [noise=true , address=45.91.116.11:4119 , piece_id=6PDH442NIXRNQCIFX2Y4PNKU5C73T2H6KNHKGMIW2UGCTC5NX7ZA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880648330 (11h33m18.469993317s) [noise=true , address=104.36.82.3:4767 , piece_id=EN3XKS2FNDR6WYC6X6WGIRXSGMT4PCJWCPEUQK7O6W5W46X4SQGQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.880718855 (11h33m18.470063842s) [noise=true , address=104.36.82.7:4910 , piece_id=VNMFNGTSUM6WKI3KIX4IMII6FV3WRMWDYTIT5T7KLJ5TRC2RCQSQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.935042530 (11h33m18.524387517s) [noise=true , address=45.91.116.5:4298 , piece_id=Y4FCHPKGJXDGWLH2FT6M5PJV5RA4UOYFQ3SHTWFJN4WO3FTFFG2A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.969979703 (11h33m18.55932468s) [noise=true , address=45.91.116.5:4299 , piece_id=5YQ5OJAPBA3N2KMYSVWQCNOTH7QPX2WCHHUXTLNSVNH2P7MZEJ3A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:18.972928735 (11h33m18.562273722s) [noise=true , address=45.91.116.12:4226 , piece_id=5QFSWOPQLZO2KEP4J27RDULUD4VCQSIE6BZ6BO3MFVFAP67OMW7A, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.047677267 (11h33m18.637022254s) [noise=true , address=104.36.82.15:4259 , piece_id=AB5IRX6ETTNZWCPBX2MKDTVTHP5OGRMECDP6ZH4HXJMFIYJVOOCA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082384121 (11h33m18.671729108s) [noise=true , address=45.91.116.4:4940 , piece_id=SPRRBU7CQUECUB5HISQOV4MFL2ZFVUSZRCCVP3FZ43L7O5XASEIQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: <nil> Aug 7 21:13:19.082409660 (11h33m18.671754647s) [seg_pos={0 0}, upload=12046]: Segment reached optimal threshold of 13 pieces. Aug 7 21:13:19.082628518 (11h33m18.671973495s) [noise=true , address=77.90.31.14:4661 , piece_id=P4DQFG2ACQKL52JQZFCTIFCUHFPLDIYEOOG4OPBJY4MIXBVKCYDQ, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Aug 7 21:13:19.082795017 (11h33m18.672139994s) [noise=true , address=104.36.82.2:4729 , piece_id=ZSUTMAAJDCCCRPJWCJBEBAQQGES4PM7CSVP4UVE75VAE3MIM6PMA, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: read tcp 188.166.225.170:56758->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082822870 (11h33m18.672167857s) [noise=true , address=104.36.82.16:4294 , piece_id=QN2G4ZPHQ2VBVMEJFPY3OLOGLHZRKEFC4SXOXSHFBQMD47QMAM7Q, seg_pos={0 0}, upload=12046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 188.166.225.170:47752->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:13:19.082839501 (11h33m18.672184478s) [seg_pos={0 0}, upload=12046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:13:19.082910688 (11h33m18.672255675s) [upload=12046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:19.082914756 (11h33m18.672259743s) [upload=12046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:19.387715025 (11h33m18.977060012s) [upload=12046]: Done uploading segments... Aug 7 21:13:19.387739612 (11h33m18.977084599s) [upload=12046]: Upload finished. err: <nil> Aug 7 21:13:45.069592425 (11h33m44.658937412s) [upload=12058]: Starting upload Aug 7 21:13:45.069646459 (11h33m44.658991446s) [upload=12058]: Done starting upload Aug 7 21:13:45.069729608 (11h33m44.659074595s) [upload=12058]: Uploading segments... Aug 7 21:13:45.069894263 (11h33m44.65923925s) [upload=12058]: Got next segment. Inline: false Aug 7 21:13:45.069927166 (11h33m44.659272153s) [upload=12058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:45.069931384 (11h33m44.659276371s) [upload=12058]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:14.428211569 (11h34m14.017556556s) [upload=12058]: Next segment err: context deadline exceeded Aug 7 21:14:14.428287775 (11h34m14.017632762s) [upload=12058]: Done uploading segments... Aug 7 21:14:14.428338853 (11h34m14.01768384s) [upload=12058]: Upload finished. err: context deadline exceeded Duration: 30.000990245s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000386556s FAIL (2025-08-07 21:14:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000403698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........XX.......................X..........................] duration: 2.010951643s FAIL (2025-08-07 21:13:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== pb.BatchRequestItem_ObjectCommit Aug 7 20:43:37.067799677 (11h3m36.657144664s) [upload=11534]: Done uploading segments... Aug 7 20:43:37.067817170 (11h3m36.657162157s) [upload=11534]: Upload finished. err: <nil> Aug 7 20:44:07.448276030 (11h4m7.037621017s) [upload=11539]: Starting upload Aug 7 20:44:07.448326927 (11h4m7.037671904s) [upload=11539]: Done starting upload Aug 7 20:44:07.448378606 (11h4m7.037723593s) [upload=11539]: Uploading segments... Aug 7 20:44:07.448408994 (11h4m7.037753981s) [upload=11539]: Got next segment. Inline: true Aug 7 20:44:07.448414504 (11h4m7.037759491s) [upload=11539]: Waiting for error group managing segments... Aug 7 20:44:07.448448068 (11h4m7.037793055s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:44:07.448451785 (11h4m7.037796762s) [upload=11539]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:44:07.448454070 (11h4m7.037799057s) [upload=11539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:44:07.799942249 (11h4m7.389287236s) [upload=11539]: Done uploading segments... Aug 7 20:44:07.799959912 (11h4m7.389304899s) [upload=11539]: Upload finished. err: <nil> Aug 7 20:45:19.440040944 (11h5m19.029385921s) [upload=11564]: Starting upload Aug 7 20:45:19.440090409 (11h5m19.029435386s) [upload=11564]: Done starting upload Aug 7 20:45:19.440146045 (11h5m19.029491032s) [upload=11564]: Uploading segments... Aug 7 20:45:19.440190359 (11h5m19.029535346s) [upload=11564]: Got next segment. Inline: true Aug 7 20:45:19.440194267 (11h5m19.029539254s) [upload=11564]: Waiting for error group managing segments... Aug 7 20:45:19.440226098 (11h5m19.029571085s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:45:19.440230026 (11h5m19.029575013s) [upload=11564]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:45:19.440232290 (11h5m19.029577277s) [upload=11564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:45:19.858443364 (11h5m19.447788352s) [upload=11564]: Done uploading segments... Aug 7 20:45:19.858467080 (11h5m19.447812067s) [upload=11564]: Upload finished. err: <nil> Aug 7 20:46:00.840349205 (11h6m0.429694192s) [upload=11575]: Starting upload Aug 7 20:46:00.840387950 (11h6m0.429732927s) [upload=11575]: Done starting upload Aug 7 20:46:00.840431102 (11h6m0.429776079s) [upload=11575]: Uploading segments... Aug 7 20:46:00.840473594 (11h6m0.429818581s) [upload=11575]: Got next segment. Inline: true Aug 7 20:46:00.840477311 (11h6m0.429822298s) [upload=11575]: Waiting for error group managing segments... Aug 7 20:46:00.840510584 (11h6m0.429855571s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:00.840514551 (11h6m0.429859539s) [upload=11575]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:00.840516746 (11h6m0.429861733s) [upload=11575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:01.149526394 (11h6m0.738871381s) [upload=11575]: Done uploading segments... Aug 7 20:46:01.149544077 (11h6m0.738889064s) [upload=11575]: Upload finished. err: <nil> Aug 7 20:46:32.531571609 (11h6m32.120916586s) [upload=11585]: Starting upload Aug 7 20:46:32.531627656 (11h6m32.120972643s) [upload=11585]: Done starting upload Aug 7 20:46:32.531679334 (11h6m32.121024311s) [upload=11585]: Uploading segments... Aug 7 20:46:32.531716416 (11h6m32.121061403s) [upload=11585]: Got next segment. Inline: true Aug 7 20:46:32.531720834 (11h6m32.121065821s) [upload=11585]: Waiting for error group managing segments... Aug 7 20:46:32.531758977 (11h6m32.121103964s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:32.531762984 (11h6m32.121107961s) [upload=11585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:32.531766161 (11h6m32.121111148s) [upload=11585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:32.823368074 (11h6m32.412713051s) [upload=11585]: Done uploading segments... Aug 7 20:46:32.823385747 (11h6m32.412730724s) [upload=11585]: Upload finished. err: <nil> Aug 7 20:46:53.991475826 (11h6m53.580820813s) [upload=11590]: Starting upload Aug 7 20:46:53.991528908 (11h6m53.580873895s) [upload=11590]: Done starting upload Aug 7 20:46:53.991580286 (11h6m53.580925273s) [upload=11590]: Uploading segments... Aug 7 20:46:53.991609822 (11h6m53.58095481s) [upload=11590]: Got next segment. Inline: true Aug 7 20:46:53.991613780 (11h6m53.580958767s) [upload=11590]: Waiting for error group managing segments... Aug 7 20:46:53.991645681 (11h6m53.580990668s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:46:53.991649578 (11h6m53.580994565s) [upload=11590]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:46:53.991651822 (11h6m53.580996809s) [upload=11590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:46:54.273031788 (11h6m53.862376835s) [upload=11590]: Done uploading segments... Aug 7 20:46:54.273056015 (11h6m53.862401032s) [upload=11590]: Upload finished. err: <nil> Aug 7 20:48:05.936480351 (11h8m5.525825338s) [upload=11610]: Starting upload Aug 7 20:48:05.936525999 (11h8m5.525870986s) [upload=11610]: Done starting upload Aug 7 20:48:05.936586324 (11h8m5.525931311s) [upload=11610]: Uploading segments... Aug 7 20:48:05.936615510 (11h8m5.525960487s) [upload=11610]: Got next segment. Inline: true Aug 7 20:48:05.936618666 (11h8m5.525963653s) [upload=11610]: Waiting for error group managing segments... Aug 7 20:48:05.936688099 (11h8m5.526033086s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:48:05.936692588 (11h8m5.526037575s) [upload=11610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:48:05.936694892 (11h8m5.526039879s) [upload=11610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:48:06.223792648 (11h8m5.813137635s) [upload=11610]: Done uploading segments... Aug 7 20:48:06.223807756 (11h8m5.813152743s) [upload=11610]: Upload finished. err: <nil> Aug 7 20:49:00.370365369 (11h8m59.959710356s) [upload=11624]: Starting upload Aug 7 20:49:00.370414633 (11h8m59.95975962s) [upload=11624]: Done starting upload Aug 7 20:49:00.370467765 (11h8m59.959812752s) [upload=11624]: Uploading segments... Aug 7 20:49:00.370501249 (11h8m59.959846236s) [upload=11624]: Got next segment. Inline: true Aug 7 20:49:00.370504986 (11h8m59.959849973s) [upload=11624]: Waiting for error group managing segments... Aug 7 20:49:00.370538900 (11h8m59.959883887s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:00.370542898 (11h8m59.959887885s) [upload=11624]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:00.370545453 (11h8m59.95989043s) [upload=11624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:00.736326238 (11h9m0.325671225s) [upload=11624]: Done uploading segments... Aug 7 20:49:00.736343821 (11h9m0.325688808s) [upload=11624]: Upload finished. err: <nil> Aug 7 20:49:43.904464651 (11h9m43.493809639s) [upload=11637]: Starting upload Aug 7 20:49:43.904540737 (11h9m43.493885724s) [upload=11637]: Done starting upload Aug 7 20:49:43.904599109 (11h9m43.493944096s) [upload=11637]: Uploading segments... Aug 7 20:49:43.904635608 (11h9m43.493980595s) [upload=11637]: Got next segment. Inline: true Aug 7 20:49:43.904639476 (11h9m43.493984463s) [upload=11637]: Waiting for error group managing segments... Aug 7 20:49:43.904675195 (11h9m43.494020182s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:49:43.904679252 (11h9m43.494024239s) [upload=11637]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:49:43.904683430 (11h9m43.494028417s) [upload=11637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:49:44.269042214 (11h9m43.858387201s) [upload=11637]: Done uploading segments... Aug 7 20:49:44.269058595 (11h9m43.858403582s) [upload=11637]: Upload finished. err: <nil> Aug 7 20:50:51.186360319 (11h10m50.775705306s) [upload=11656]: Starting upload Aug 7 20:50:51.186464308 (11h10m50.775809295s) [upload=11656]: Done starting upload Aug 7 20:50:51.186527319 (11h10m50.775872306s) [upload=11656]: Uploading segments... Aug 7 20:50:51.186558759 (11h10m50.775903746s) [upload=11656]: Got next segment. Inline: true Aug 7 20:50:51.186562295 (11h10m50.775907282s) [upload=11656]: Waiting for error group managing segments... Aug 7 20:50:51.186595108 (11h10m50.775940095s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:50:51.186599105 (11h10m50.775944082s) [upload=11656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:50:51.186601591 (11h10m50.775946578s) [upload=11656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:50:51.466357188 (11h10m51.055702175s) [upload=11656]: Done uploading segments... Aug 7 20:50:51.466375784 (11h10m51.055720781s) [upload=11656]: Upload finished. err: <nil> Aug 7 20:51:54.183767656 (11h11m53.773112643s) [upload=11674]: Starting upload Aug 7 20:51:54.183810869 (11h11m53.773155846s) [upload=11674]: Done starting upload Aug 7 20:51:54.183850504 (11h11m53.773195481s) [upload=11674]: Uploading segments... Aug 7 20:51:54.183877035 (11h11m53.773222012s) [upload=11674]: Got next segment. Inline: true Aug 7 20:51:54.183880662 (11h11m53.773225639s) [upload=11674]: Waiting for error group managing segments... Aug 7 20:51:54.183910529 (11h11m53.773255516s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 20:51:54.183914426 (11h11m53.773259413s) [upload=11674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 20:51:54.183916961 (11h11m53.773261938s) [upload=11674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 20:51:54.662596438 (11h11m54.251941425s) [upload=11674]: Done uploading segments... Aug 7 20:51:54.662619702 (11h11m54.251964689s) [upload=11674]: Upload finished. err: <nil> Duration:* 2.011022429s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.961420658s FAIL (2025-08-07 21:13:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.961447017s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001545206s FAIL (2025-08-07 21:13:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 58.234887219 (11h26m57.824232206s) [upload=11935]: Upload finished. err: <nil> Aug 7 21:07:27.109267973 (11h27m26.69861301s) [upload=11945]: Starting upload Aug 7 21:07:27.109416217 (11h27m26.698761254s) [upload=11945]: Done starting upload Aug 7 21:07:27.109513132 (11h27m26.698858119s) [upload=11945]: Uploading segments... Aug 7 21:07:27.109584840 (11h27m26.698929827s) [upload=11945]: Got next segment. Inline: true Aug 7 21:07:27.109589388 (11h27m26.698934375s) [upload=11945]: Waiting for error group managing segments... Aug 7 21:07:27.109631449 (11h27m26.698976436s) [upload=11945]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:07:27.109636138 (11h27m26.698981125s) [upload=11945]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:07:27.109638613 (11h27m26.6989836s) [upload=11945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:07:27.394851845 (11h27m26.984196832s) [upload=11945]: Done uploading segments... Aug 7 21:07:27.394867715 (11h27m26.984212702s) [upload=11945]: Upload finished. err: <nil> Aug 7 21:08:20.470385011 (11h28m20.059729988s) [upload=11961]: Starting upload Aug 7 21:08:20.470444956 (11h28m20.059789943s) [upload=11961]: Done starting upload Aug 7 21:08:20.470517715 (11h28m20.059862702s) [upload=11961]: Uploading segments... Aug 7 21:08:20.470590324 (11h28m20.059935311s) [upload=11961]: Got next segment. Inline: true Aug 7 21:08:20.470594982 (11h28m20.059939971s) [upload=11961]: Waiting for error group managing segments... Aug 7 21:08:20.470639418 (11h28m20.059984405s) [upload=11961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:08:20.470645349 (11h28m20.059990336s) [upload=11961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:08:20.470647794 (11h28m20.059992781s) [upload=11961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:08:20.752099520 (11h28m20.341444507s) [upload=11961]: Done uploading segments... Aug 7 21:08:20.752117565 (11h28m20.341462552s) [upload=11961]: Upload finished. err: <nil> Aug 7 21:09:25.498062703 (11h29m25.08740769s) [upload=11978]: Starting upload Aug 7 21:09:25.498115193 (11h29m25.08746018s) [upload=11978]: Done starting upload Aug 7 21:09:25.498162373 (11h29m25.08750736s) [upload=11978]: Uploading segments... Aug 7 21:09:25.498190667 (11h29m25.087535654s) [upload=11978]: Got next segment. Inline: true Aug 7 21:09:25.498196007 (11h29m25.087540994s) [upload=11978]: Waiting for error group managing segments... Aug 7 21:09:25.498229983 (11h29m25.08757497s) [upload=11978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:25.498233650 (11h29m25.087578637s) [upload=11978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:25.498236134 (11h29m25.087581121s) [upload=11978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:25.808420372 (11h29m25.397765359s) [upload=11978]: Done uploading segments... Aug 7 21:09:25.808436974 (11h29m25.397781961s) [upload=11978]: Upload finished. err: <nil> Aug 7 21:09:47.139455442 (11h29m46.728800429s) [upload=11988]: Starting upload Aug 7 21:09:47.139503314 (11h29m46.728848301s) [upload=11988]: Done starting upload Aug 7 21:09:47.139548831 (11h29m46.728893818s) [upload=11988]: Uploading segments... Aug 7 21:09:47.139579450 (11h29m46.728924437s) [upload=11988]: Got next segment. Inline: true Aug 7 21:09:47.139583277 (11h29m46.728928264s) [upload=11988]: Waiting for error group managing segments... Aug 7 21:09:47.139618274 (11h29m46.728963261s) [upload=11988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:47.139622451 (11h29m46.728967439s) [upload=11988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:47.139624797 (11h29m46.728969784s) [upload=11988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.434297548 (11h29m47.023642535s) [upload=11988]: Done uploading segments... Aug 7 21:09:47.434326894 (11h29m47.023671872s) [upload=11988]: Upload finished. err: <nil> Aug 7 21:10:17.433462616 (11h30m17.022807593s) [upload=11995]: Starting upload Aug 7 21:10:17.433520777 (11h30m17.022865764s) [upload=11995]: Done starting upload Aug 7 21:10:17.433584859 (11h30m17.022929846s) [upload=11995]: Uploading segments... Aug 7 21:10:17.433692035 (11h30m17.023037012s) [upload=11995]: Got next segment. Inline: true Aug 7 21:10:17.433696885 (11h30m17.023041872s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:10:17.433744876 (11h30m17.023089853s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:17.433749755 (11h30m17.023094742s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:17.433754204 (11h30m17.023099191s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:17.710858662 (11h30m17.300203709s) [upload=11995]: Done uploading segments... Aug 7 21:10:17.710881997 (11h30m17.300227024s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:10:36.218517401 (11h30m35.807862378s) [upload=11998]: Starting upload Aug 7 21:10:36.218649954 (11h30m35.807994941s) [upload=11998]: Done starting upload Aug 7 21:10:36.218713896 (11h30m35.808058913s) [upload=11998]: Uploading segments... Aug 7 21:10:36.218766106 (11h30m35.808111103s) [upload=11998]: Got next segment. Inline: true Aug 7 21:10:36.218772398 (11h30m35.808117425s) [upload=11998]: Waiting for error group managing segments... Aug 7 21:10:36.218820571 (11h30m35.808165558s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:36.218825550 (11h30m35.808170537s) [upload=11998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:36.218828946 (11h30m35.808173943s) [upload=11998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:36.494741534 (11h30m36.084086521s) [upload=11998]: Done uploading segments... Aug 7 21:10:36.494760030 (11h30m36.084105007s) [upload=11998]: Upload finished. err: <nil> Aug 7 21:11:36.918559943 (11h31m36.50790493s) [upload=12018]: Starting upload Aug 7 21:11:36.918615830 (11h31m36.507960817s) [upload=12018]: Done starting upload Aug 7 21:11:36.918660295 (11h31m36.508005282s) [upload=12018]: Uploading segments... Aug 7 21:11:36.918688629 (11h31m36.508033606s) [upload=12018]: Got next segment. Inline: true Aug 7 21:11:36.918691745 (11h31m36.508036732s) [upload=12018]: Waiting for error group managing segments... Aug 7 21:11:36.918721552 (11h31m36.508066539s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:36.918725530 (11h31m36.508070517s) [upload=12018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:36.918727794 (11h31m36.508072781s) [upload=12018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:37.211063805 (11h31m36.800408792s) [upload=12018]: Done uploading segments... Aug 7 21:11:37.211087029 (11h31m36.800432026s) [upload=12018]: Upload finished. err: <nil> Aug 7 21:12:15.260577538 (11h32m14.849922525s) [upload=12032]: Starting upload Aug 7 21:12:15.260632023 (11h32m14.84997701s) [upload=12032]: Done starting upload Aug 7 21:12:15.260678571 (11h32m14.850023548s) [upload=12032]: Uploading segments... Aug 7 21:12:15.260706434 (11h32m14.850051421s) [upload=12032]: Got next segment. Inline: true Aug 7 21:12:15.260710011 (11h32m14.850054998s) [upload=12032]: Waiting for error group managing segments... Aug 7 21:12:15.260743897 (11h32m14.850088874s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:15.260747644 (11h32m14.850092631s) [upload=12032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:15.260750389 (11h32m14.850095376s) [upload=12032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:15.565323071 (11h32m15.154668058s) [upload=12032]: Done uploading segments... Aug 7 21:12:15.565341296 (11h32m15.154686283s) [upload=12032]: Upload finished. err: <nil> Aug 7 21:12:41.001560374 (11h32m40.590905361s) [upload=12034]: Starting upload Aug 7 21:12:41.001659143 (11h32m40.59100413s) [upload=12034]: Done starting upload Aug 7 21:12:41.001745669 (11h32m40.591090666s) [upload=12034]: Uploading segments... Aug 7 21:12:41.001852704 (11h32m40.591197701s) [upload=12034]: Got next segment. Inline: true Aug 7 21:12:41.001857372 (11h32m40.591202359s) [upload=12034]: Waiting for error group managing segments... Aug 7 21:12:41.001907849 (11h32m40.591252836s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:41.001913169 (11h32m40.591258156s) [upload=12034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:41.001915584 (11h32m40.591260571s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:41.286244505 (11h32m40.875589493s) [upload=12034]: Done uploading segments... Aug 7 21:12:41.286260567 (11h32m40.875605544s) [upload=12034]: Upload finished. err: <nil> Aug 7 21:13:16.624584146 (11h33m16.213929123s) [upload=12047]: Starting upload Aug 7 21:13:16.624659630 (11h33m16.214004607s) [upload=12047]: Done starting upload Aug 7 21:13:16.624710127 (11h33m16.214055114s) [upload=12047]: Uploading segments... Aug 7 21:13:16.624745715 (11h33m16.214090702s) [upload=12047]: Got next segment. Inline: true Aug 7 21:13:16.624750654 (11h33m16.214095641s) [upload=12047]: Waiting for error group managing segments... Aug 7 21:13:16.624782977 (11h33m16.214127964s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:16.624786804 (11h33m16.214131791s) [upload=12047]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:16.624789339 (11h33m16.214134326s) [upload=12047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.974391598 (11h33m45.563736596s) [upload=12047]: Done uploading segments... Aug 7 21:13:45.974419923 (11h33m45.56376491s) [upload=12047]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001577498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XX...........X.......................X..] duration: 6.311971871s FAIL (2025-08-07 21:13:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 7106407 (11h31m46.936451394s) [upload=12022]: Uploading segments... Aug 7 21:11:47.347198184 (11h31m46.936543161s) [upload=12022]: Got next segment. Inline: false Aug 7 21:11:47.347234122 (11h31m46.936579109s) [upload=12022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:47.347237969 (11h31m46.936582956s) [upload=12022]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:11:47.802049075 (11h31m47.391394053s) [seg_pos={0 0}, upload=12022]: Begin upload segment... Aug 7 21:11:47.802118007 (11h31m47.391462994s) [seg_pos={0 0}, upload=12022]: Done begin upload segment. Aug 7 21:11:47.802142414 (11h31m47.391487401s) [noise=true , address=104.36.82.22:4541 , piece_id=Q5X72MXPD2VX3TRLK37VB3OV5Y4WLD4P6CWPAVL2X4D6BSIIOFSQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802186428 (11h31m47.391531415s) [noise=true , address=104.36.82.5:4846 , piece_id=6NNDPTLCBYLGFWU2H57AIRLEEXTREPMET5R2YZETWQYPIP2ZUHHQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802550154 (11h31m47.391895141s) [noise=true , address=104.36.82.46:4501 , piece_id=CVPKLEXG5FMF3NAR3ONF7LGIWUXJ7OSF5BBOTJ3BLQE4CFFNQKBA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802702837 (11h31m47.392047814s) [noise=true , address=45.91.116.9:4268 , piece_id=VJP2ZZ3UMB2XMXGDGIV45YOLKHRNNJORX2NIB3SHBCI4UZNJEN7A, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.802844608 (11h31m47.392189595s) [noise=true , address=104.36.82.16:4328 , piece_id=BFL3YNCSEKYNXJV6PJDPERHGUC6PDQZ52JNSLZDZPH3DCLYCAJPA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803081390 (11h31m47.392426377s) [noise=true , address=104.36.82.47:4523 , piece_id=4WH6OVD7YSUMLPUBHAQPDR5VYX7Y36WKS3MSM7ULY6CI73BALULQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803116328 (11h31m47.392461315s) [noise=true , address=45.91.116.11:4150 , piece_id=64KLQELYVF4SAJJYCFSQD3X5USFFYLXGNOJOQGZ36LOYOFEJ2UBQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803402735 (11h31m47.392747722s) [noise=true , address=104.36.82.12:4165 , piece_id=BBVGOIQK3IBDXSA5KXGHV4Z4TRPZYJQGJT3PYQJXZUI7ZJ55KRJA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803414909 (11h31m47.392759896s) [noise=true , address=104.36.82.39:4752 , piece_id=ITMK3WWPOHUBVDUV2ZUL65XLBGCIVZKGUD544R2K3FCAMTQN5TXQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803693041 (11h31m47.393038028s) [noise=true , address=104.36.82.36:4622 , piece_id=JYCNDCLZYJWGEB4ZWJM5ES3G25WPLBPOY6UGXI37SFC467KTIZEA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.803929323 (11h31m47.39327431s) [noise=true , address=104.36.82.42:4813 , piece_id=UAXL4ZUQ4W6P3DLPKIWW7KC3U2MZPZQ25ZA7Z4A7BFQ42EMJZAXQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804163892 (11h31m47.393508879s) [noise=true , address=104.36.82.58:4269 , piece_id=QX6CM6DHSIW2KXY2Z6NRMSRG3HMBRNBQ3JRPHXBGOQMEL5HVZ5ZQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804383622 (11h31m47.393728609s) [noise=true , address=45.91.116.4:4969 , piece_id=GVQDU7BRCLA6PHFSRGOJZBAV6QZQRI2ZEG3EM6I2TJAZ4JAE4P7Q, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804621557 (11h31m47.393966544s) [noise=true , address=104.36.82.23:4453 , piece_id=BLRIGJ5Z3NNUPUHC2YZW4PDMK2SDUVPE75GO5T4WSFTT2FDOEEOQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804837270 (11h31m47.394182257s) [noise=true , address=104.36.82.2:4744 , piece_id=V6TRPHHDLGWKVLQH7BT6AVYVOKVVP3EHL67D62SPHAKNMWICYJWQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.804911372 (11h31m47.394256359s) [noise=true , address=104.36.82.43:4849 , piece_id=IZH45DQEPVT7WEQ2UDAAA7OS37XKMKG4VBJRRUEKQ3G36YUM2LMQ, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.805071538 (11h31m47.394416515s) [noise=true , address=64.49.13.170:4728 , piece_id=LED5VGYENQQBGOP3UH4NKT6O6TH4M6OEZCIXZ5N7AYWFLW452KSA, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:47.805261793 (11h31m47.39460677s) [noise=true , address=104.36.82.44:4899 , piece_id=I5PVQ6WHWORI2TFF2SJSZDAUDH3FX5DVNPP23VC6JECMDAO6ZM6Q, seg_pos={0 0}, upload=12022]: Uploading piece... Aug 7 21:11:48.969580099 (11h31m48.558925086s) [upload=12022]: Next returned nil segment Aug 7 21:11:48.969596991 (11h31m48.558941978s) [upload=12022]: Waiting for error group managing segments... Aug 7 21:11:49.265222255 (11h31m48.854567242s) [noise=true , address=64.49.13.170:4728 , piece_id=LED5VGYENQQBGOP3UH4NKT6O6TH4M6OEZCIXZ5N7AYWFLW452KSA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.737280759 (11h31m49.326625746s) [noise=true , address=104.36.82.58:4269 , piece_id=QX6CM6DHSIW2KXY2Z6NRMSRG3HMBRNBQ3JRPHXBGOQMEL5HVZ5ZQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.753249862 (11h31m49.342594849s) [noise=true , address=104.36.82.12:4165 , piece_id=BBVGOIQK3IBDXSA5KXGHV4Z4TRPZYJQGJT3PYQJXZUI7ZJ55KRJA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.780910703 (11h31m49.37025569s) [noise=true , address=45.91.116.9:4268 , piece_id=VJP2ZZ3UMB2XMXGDGIV45YOLKHRNNJORX2NIB3SHBCI4UZNJEN7A, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.813098015 (11h31m49.402442992s) [noise=true , address=104.36.82.46:4501 , piece_id=CVPKLEXG5FMF3NAR3ONF7LGIWUXJ7OSF5BBOTJ3BLQE4CFFNQKBA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.827616893 (11h31m49.41696188s) [noise=true , address=104.36.82.42:4813 , piece_id=UAXL4ZUQ4W6P3DLPKIWW7KC3U2MZPZQ25ZA7Z4A7BFQ42EMJZAXQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.844644011 (11h31m49.433988998s) [noise=true , address=104.36.82.5:4846 , piece_id=6NNDPTLCBYLGFWU2H57AIRLEEXTREPMET5R2YZETWQYPIP2ZUHHQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.896979020 (11h31m49.486324007s) [noise=true , address=104.36.82.16:4328 , piece_id=BFL3YNCSEKYNXJV6PJDPERHGUC6PDQZ52JNSLZDZPH3DCLYCAJPA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.913302922 (11h31m49.502647909s) [noise=true , address=104.36.82.36:4622 , piece_id=JYCNDCLZYJWGEB4ZWJM5ES3G25WPLBPOY6UGXI37SFC467KTIZEA, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.928574831 (11h31m49.517919809s) [noise=true , address=104.36.82.22:4541 , piece_id=Q5X72MXPD2VX3TRLK37VB3OV5Y4WLD4P6CWPAVL2X4D6BSIIOFSQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:49.967608171 (11h31m49.556953148s) [noise=true , address=104.36.82.43:4849 , piece_id=IZH45DQEPVT7WEQ2UDAAA7OS37XKMKG4VBJRRUEKQ3G36YUM2LMQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.087000568 (11h31m49.676345555s) [noise=true , address=45.91.116.11:4150 , piece_id=64KLQELYVF4SAJJYCFSQD3X5USFFYLXGNOJOQGZ36LOYOFEJ2UBQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.098037264 (11h31m49.687382241s) [noise=true , address=45.91.116.4:4969 , piece_id=GVQDU7BRCLA6PHFSRGOJZBAV6QZQRI2ZEG3EM6I2TJAZ4JAE4P7Q, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.160148408 (11h31m49.749493395s) [noise=true , address=104.36.82.47:4523 , piece_id=4WH6OVD7YSUMLPUBHAQPDR5VYX7Y36WKS3MSM7ULY6CI73BALULQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: <nil> Aug 7 21:11:50.160172064 (11h31m49.749517051s) [seg_pos={0 0}, upload=12022]: Segment reached optimal threshold of 14 pieces. Aug 7 21:11:50.160367668 (11h31m49.749712655s) [noise=true , address=104.36.82.44:4899 , piece_id=I5PVQ6WHWORI2TFF2SJSZDAUDH3FX5DVNPP23VC6JECMDAO6ZM6Q, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.160428794 (11h31m49.749773782s) [noise=true , address=104.36.82.23:4453 , piece_id=BLRIGJ5Z3NNUPUHC2YZW4PDMK2SDUVPE75GO5T4WSFTT2FDOEEOQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.160601615 (11h31m49.749946592s) [noise=true , address=104.36.82.2:4744 , piece_id=V6TRPHHDLGWKVLQH7BT6AVYVOKVVP3EHL67D62SPHAKNMWICYJWQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: read tcp 188.166.225.170:55148->104.36.82.2:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:11:50.160658343 (11h31m49.75000333s) [noise=true , address=104.36.82.39:4752 , piece_id=ITMK3WWPOHUBVDUV2ZUL65XLBGCIVZKGUD544R2K3FCAMTQN5TXQ, seg_pos={0 0}, upload=12022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: read tcp 188.166.225.170:46618->104.36.82.39:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:11:50.160688220 (11h31m49.750033207s) [seg_pos={0 0}, upload=12022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:11:50.160746132 (11h31m49.750091179s) [upload=12022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:11:50.160752644 (11h31m49.750097651s) [upload=12022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:50.684435488 (11h31m50.273780475s) [upload=12022]: Done uploading segments... Aug 7 21:11:50.684472599 (11h31m50.273817576s) [upload=12022]: Upload finished. err: <nil> Duration: 6.311996879s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.194083s FAIL (2025-08-07 21:13:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 21:11:35.434526944 (11h31m35.023871921s) [upload=12017]: Done starting upload Aug 7 21:11:35.434621335 (11h31m35.023966362s) [upload=12017]: Uploading segments... Aug 7 21:11:35.434728259 (11h31m35.024073246s) [upload=12017]: Got next segment. Inline: false Aug 7 21:11:35.434762164 (11h31m35.024107151s) [upload=12017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:35.434766152 (11h31m35.024111139s) [upload=12017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:11:35.668275282 (11h31m35.257620259s) [seg_pos={0 0}, upload=12017]: Begin upload segment... Aug 7 21:11:35.668321981 (11h31m35.257666968s) [seg_pos={0 0}, upload=12017]: Done begin upload segment. Aug 7 21:11:35.668348001 (11h31m35.257692988s) [noise=true , address=45.91.116.7:4011 , piece_id=337XKW3GUCNSR6BLKFBPHU4IJUKJH6DYCBZW7ZXJ45SU6S2UZ2LA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.668411813 (11h31m35.2577568s) [noise=true , address=77.90.31.19:4843 , piece_id=AEUFO6KWQTAFBIQNKWDNQ3EL3XC2EKA7ULMHFYASCRS42YQMDLAA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.668705525 (11h31m35.258050512s) [noise=true , address=104.36.82.16:4294 , piece_id=JNKVRT2O7XLGKO5IHUIZCYAPKZ3SCABTR6FFS3M7GPSN7EBLQ4DQ, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.668877424 (11h31m35.258222401s) [noise=true , address=77.90.31.14:4662 , piece_id=X6U7XYH6AQHXARYULW2HSECW64RKAQNIREF3AAKJEK5COFZLLQ2A, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.668970081 (11h31m35.258315068s) [noise=true , address=77.90.31.10:4517 , piece_id=WPBIYBERJSHAGXUKBVSGJREJ6B37MNBOHNYR5GYRSX5N52G2RJ5A, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669186065 (11h31m35.258531052s) [noise=true , address=104.36.82.6:4876 , piece_id=OYGIZ34JUDMX64BFSPRTQIHKCW6ENRCQEYHE7QPNNS2BMPOR7M4Q, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669276127 (11h31m35.258621114s) [noise=true , address=45.91.116.14:4190 , piece_id=VRHZH224IVJRPT5P7C6S5EFPWVTUWZMEURBNSBPZUAONVVUERS3A, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669522218 (11h31m35.258867205s) [noise=true , address=45.91.116.8:4045 , piece_id=PX2WHDM4EYXXB3QZXUFDVEYNI23O5MF5N7UGUMLJY27DTI3L3LMA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669566662 (11h31m35.258911649s) [noise=true , address=77.90.31.16:4733 , piece_id=764U6SFKPDXTZ2LABZAIIMISMPJOBOE6TDZ734OWSFKSVFPVDBZQ, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669830978 (11h31m35.259175965s) [noise=true , address=45.91.116.10:4081 , piece_id=2P5ZU46HRSVSGTAM7R3G3B6PGYATVURICNCAM5R5YPKY44IFKX3Q, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669869942 (11h31m35.259214929s) [noise=true , address=104.36.82.9:4005 , piece_id=LFBGDFRAYCSOBC6C47OHXPUIXSDEWL3HNAG2H4M25L3IHU3KGHJQ, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670054415 (11h31m35.259399393s) [noise=true , address=104.36.82.14:4223 , piece_id=4KR6G7VECDZRUMWFPYBXSK4IK27X6YXRSEGOPPLSFWRDTQEM66YA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670199142 (11h31m35.259544129s) [noise=true , address=45.91.116.6:4973 , piece_id=7MFZLVPY2BFFGMEIBBHTT6UUCFRYWQNDFRDEIN7WULORX37C2EWA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670281369 (11h31m35.259626357s) [noise=true , address=45.91.116.13:4153 , piece_id=JNIPJGHRPEJFGTZDBSOLPFPI6FNH23VWBO2O4DVFKOCUROKBAMQQ, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670499076 (11h31m35.259844053s) [noise=true , address=45.91.116.14:4191 , piece_id=64X5WB6NWW7QSOBVMO257IFCOLHCKIL6C3G7EMPVCR5PBMN2XPVA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670765476 (11h31m35.260110453s) [noise=true , address=45.91.116.9:4262 , piece_id=R4TTQXN2BJKQ5WQ734B7WMDXZ3C4NAKBIQECI24WEZR3ZUSRPZEA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:37.305581962 (11h31m36.894926949s) [upload=12017]: Next returned nil segment Aug 7 21:11:37.305598983 (11h31m36.894943961s) [upload=12017]: Waiting for error group managing segments... Aug 7 21:11:37.704891137 (11h31m37.294236135s) [noise=true , address=45.91.116.10:4081 , piece_id=2P5ZU46HRSVSGTAM7R3G3B6PGYATVURICNCAM5R5YPKY44IFKX3Q, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.722340423 (11h31m37.31168541s) [noise=true , address=104.36.82.16:4294 , piece_id=JNKVRT2O7XLGKO5IHUIZCYAPKZ3SCABTR6FFS3M7GPSN7EBLQ4DQ, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.759508254 (11h31m37.348853241s) [noise=true , address=77.90.31.19:4843 , piece_id=AEUFO6KWQTAFBIQNKWDNQ3EL3XC2EKA7ULMHFYASCRS42YQMDLAA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.774868541 (11h31m37.364213528s) [noise=true , address=77.90.31.10:4517 , piece_id=WPBIYBERJSHAGXUKBVSGJREJ6B37MNBOHNYR5GYRSX5N52G2RJ5A, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.780756907 (11h31m37.370101894s) [noise=true , address=45.91.116.7:4011 , piece_id=337XKW3GUCNSR6BLKFBPHU4IJUKJH6DYCBZW7ZXJ45SU6S2UZ2LA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.816266847 (11h31m37.405611834s) [noise=true , address=104.36.82.6:4876 , piece_id=OYGIZ34JUDMX64BFSPRTQIHKCW6ENRCQEYHE7QPNNS2BMPOR7M4Q, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.828575126 (11h31m37.417920113s) [noise=true , address=77.90.31.16:4733 , piece_id=764U6SFKPDXTZ2LABZAIIMISMPJOBOE6TDZ734OWSFKSVFPVDBZQ, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.844647896 (11h31m37.433992883s) [noise=true , address=45.91.116.8:4045 , piece_id=PX2WHDM4EYXXB3QZXUFDVEYNI23O5MF5N7UGUMLJY27DTI3L3LMA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.867042233 (11h31m37.45638723s) [noise=true , address=104.36.82.14:4223 , piece_id=4KR6G7VECDZRUMWFPYBXSK4IK27X6YXRSEGOPPLSFWRDTQEM66YA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.888714938 (11h31m37.478059925s) [noise=true , address=45.91.116.6:4973 , piece_id=7MFZLVPY2BFFGMEIBBHTT6UUCFRYWQNDFRDEIN7WULORX37C2EWA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.963433603 (11h31m37.55277859s) [noise=true , address=45.91.116.13:4153 , piece_id=JNIPJGHRPEJFGTZDBSOLPFPI6FNH23VWBO2O4DVFKOCUROKBAMQQ, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.986126871 (11h31m37.575471858s) [noise=true , address=45.91.116.14:4191 , piece_id=64X5WB6NWW7QSOBVMO257IFCOLHCKIL6C3G7EMPVCR5PBMN2XPVA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:38.073190565 (11h31m37.662535552s) [noise=true , address=77.90.31.14:4662 , piece_id=X6U7XYH6AQHXARYULW2HSECW64RKAQNIREF3AAKJEK5COFZLLQ2A, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:38.073212426 (11h31m37.662557413s) [seg_pos={0 0}, upload=12017]: Segment reached optimal threshold of 13 pieces. Aug 7 21:11:38.073319682 (11h31m37.662664669s) [noise=true , address=45.91.116.9:4262 , piece_id=R4TTQXN2BJKQ5WQ734B7WMDXZ3C4NAKBIQECI24WEZR3ZUSRPZEA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:38.073386219 (11h31m37.662731207s) [noise=true , address=45.91.116.14:4190 , piece_id=VRHZH224IVJRPT5P7C6S5EFPWVTUWZMEURBNSBPZUAONVVUERS3A, seg_pos={0 0}, upload=12017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:38.073506179 (11h31m37.662851166s) [noise=true , address=104.36.82.9:4005 , piece_id=LFBGDFRAYCSOBC6C47OHXPUIXSDEWL3HNAG2H4M25L3IHU3KGHJQ, seg_pos={0 0}, upload=12017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: write tcp 188.166.225.170:40678->104.36.82.9:4005: use of closed network connection; write tcp 188.166.225.170:40678->104.36.82.9:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40678->104.36.82.9:4005: use of closed network connection Aug 7 21:11:38.073542368 (11h31m37.662887355s) [seg_pos={0 0}, upload=12017]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:11:38.073602473 (11h31m37.66294745s) [upload=12017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:11:38.073634404 (11h31m37.662979391s) [upload=12017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:38.377251317 (11h31m37.966596304s) [upload=12017]: Done uploading segments... Aug 7 21:11:38.377276456 (11h31m37.966621433s) [upload=12017]: Upload finished. err: <nil> Aug 7 21:12:29.590155214 (11h32m29.179500201s) [upload=12033]: Starting upload Aug 7 21:12:29.590208055 (11h32m29.179553042s) [upload=12033]: Done starting upload Aug 7 21:12:29.590306493 (11h32m29.17965148s) [upload=12033]: Uploading segments... Aug 7 21:12:29.590377449 (11h32m29.179722436s) [upload=12033]: Got next segment. Inline: false Aug 7 21:12:29.590398359 (11h32m29.179743346s) [upload=12033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:29.590401966 (11h32m29.179746953s) [upload=12033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:12:58.939555566 (11h32m58.528900553s) [upload=12033]: Next segment err: context deadline exceeded Aug 7 21:12:58.939613887 (11h32m58.528958874s) [upload=12033]: Done uploading segments... Aug 7 21:12:58.939626821 (11h32m58.528971798s) [upload=12033]: Upload finished. err: context deadline exceeded Duration: 6.194108248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.97903417s FAIL (2025-08-07 21:13:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5959385 (11h24m50.225304372s) [upload=11893]: Upload finished. err: <nil> Aug 7 21:05:25.355558074 (11h25m24.944903051s) [upload=11908]: Starting upload Aug 7 21:05:25.355615374 (11h25m24.944960361s) [upload=11908]: Done starting upload Aug 7 21:05:25.355660210 (11h25m24.945005197s) [upload=11908]: Uploading segments... Aug 7 21:05:25.355689586 (11h25m24.945034563s) [upload=11908]: Got next segment. Inline: true Aug 7 21:05:25.355693293 (11h25m24.94503828s) [upload=11908]: Waiting for error group managing segments... Aug 7 21:05:25.355727409 (11h25m24.945072396s) [upload=11908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:05:25.355731426 (11h25m24.945076404s) [upload=11908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:05:25.355733941 (11h25m24.945078928s) [upload=11908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:05:25.663989869 (11h25m25.253334856s) [upload=11908]: Done uploading segments... Aug 7 21:05:25.664007443 (11h25m25.25335242s) [upload=11908]: Upload finished. err: <nil> Aug 7 21:05:47.945556961 (11h25m47.534901948s) [upload=11916]: Starting upload Aug 7 21:05:47.945614611 (11h25m47.534959598s) [upload=11916]: Done starting upload Aug 7 21:05:47.945674546 (11h25m47.535019533s) [upload=11916]: Uploading segments... Aug 7 21:05:47.945704424 (11h25m47.535049411s) [upload=11916]: Got next segment. Inline: true Aug 7 21:05:47.945708231 (11h25m47.535053208s) [upload=11916]: Waiting for error group managing segments... Aug 7 21:05:47.945742226 (11h25m47.535087213s) [upload=11916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:05:47.945746303 (11h25m47.53509129s) [upload=11916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:05:47.945748708 (11h25m47.535093695s) [upload=11916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:05:48.221309343 (11h25m47.81065433s) [upload=11916]: Done uploading segments... Aug 7 21:05:48.221332277 (11h25m47.810677304s) [upload=11916]: Upload finished. err: <nil> Aug 7 21:06:29.023021686 (11h26m28.612366673s) [upload=11927]: Starting upload Aug 7 21:06:29.023068907 (11h26m28.612413894s) [upload=11927]: Done starting upload Aug 7 21:06:29.023149912 (11h26m28.612494899s) [upload=11927]: Uploading segments... Aug 7 21:06:29.023185480 (11h26m28.612530457s) [upload=11927]: Got next segment. Inline: true Aug 7 21:06:29.023189928 (11h26m28.612534915s) [upload=11927]: Waiting for error group managing segments... Aug 7 21:06:29.023231568 (11h26m28.612576555s) [upload=11927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:06:29.023236387 (11h26m28.612581374s) [upload=11927]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:06:29.023239593 (11h26m28.61258458s) [upload=11927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:06:29.300861462 (11h26m28.890206449s) [upload=11927]: Done uploading segments... Aug 7 21:06:29.300881371 (11h26m28.890226358s) [upload=11927]: Upload finished. err: <nil> Aug 7 21:07:37.321661864 (11h27m36.911006841s) [upload=11949]: Starting upload Aug 7 21:07:37.321718112 (11h27m36.911063099s) [upload=11949]: Done starting upload Aug 7 21:07:37.321771434 (11h27m36.911116411s) [upload=11949]: Uploading segments... Aug 7 21:07:37.321872968 (11h27m36.911217945s) [upload=11949]: Got next segment. Inline: true Aug 7 21:07:37.321877877 (11h27m36.911222865s) [upload=11949]: Waiting for error group managing segments... Aug 7 21:07:37.321923275 (11h27m36.911268262s) [upload=11949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:07:37.321928314 (11h27m36.911273301s) [upload=11949]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:07:37.321930418 (11h27m36.911275405s) [upload=11949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:07:37.599927314 (11h27m37.189272301s) [upload=11949]: Done uploading segments... Aug 7 21:07:37.599948635 (11h27m37.189293672s) [upload=11949]: Upload finished. err: <nil> Aug 7 21:08:35.851762080 (11h28m35.441107057s) [upload=11967]: Starting upload Aug 7 21:08:35.851828216 (11h28m35.441173203s) [upload=11967]: Done starting upload Aug 7 21:08:35.851888952 (11h28m35.441233939s) [upload=11967]: Uploading segments... Aug 7 21:08:35.851977722 (11h28m35.441322709s) [upload=11967]: Got next segment. Inline: true Aug 7 21:08:35.851983424 (11h28m35.441328411s) [upload=11967]: Waiting for error group managing segments... Aug 7 21:08:35.852017519 (11h28m35.441362506s) [upload=11967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:08:35.852021105 (11h28m35.441366093s) [upload=11967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:08:35.852023460 (11h28m35.441368447s) [upload=11967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:08:36.115286628 (11h28m35.704631615s) [upload=11967]: Done uploading segments... Aug 7 21:08:36.115331112 (11h28m35.704676099s) [upload=11967]: Upload finished. err: <nil> Aug 7 21:09:20.788493181 (11h29m20.377838178s) [upload=11976]: Starting upload Aug 7 21:09:20.788595918 (11h29m20.377940965s) [upload=11976]: Done starting upload Aug 7 21:09:20.788684948 (11h29m20.378029975s) [upload=11976]: Uploading segments... Aug 7 21:09:20.788747948 (11h29m20.378092935s) [upload=11976]: Got next segment. Inline: true Aug 7 21:09:20.788756344 (11h29m20.378101361s) [upload=11976]: Waiting for error group managing segments... Aug 7 21:09:20.788807752 (11h29m20.378152739s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:20.788813985 (11h29m20.378158972s) [upload=11976]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:20.788818584 (11h29m20.378163611s) [upload=11976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:21.070630248 (11h29m20.659975235s) [upload=11976]: Done uploading segments... Aug 7 21:09:21.070652190 (11h29m20.659997247s) [upload=11976]: Upload finished. err: <nil> Aug 7 21:09:46.948530536 (11h29m46.537875523s) [upload=11987]: Starting upload Aug 7 21:09:46.948588607 (11h29m46.537933584s) [upload=11987]: Done starting upload Aug 7 21:09:46.948642600 (11h29m46.537987587s) [upload=11987]: Uploading segments... Aug 7 21:09:46.948702786 (11h29m46.538047773s) [upload=11987]: Got next segment. Inline: true Aug 7 21:09:46.948707776 (11h29m46.538052763s) [upload=11987]: Waiting for error group managing segments... Aug 7 21:09:46.948747241 (11h29m46.538092228s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:46.948752340 (11h29m46.538097327s) [upload=11987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:46.948754896 (11h29m46.538099883s) [upload=11987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:47.214261274 (11h29m46.803606261s) [upload=11987]: Done uploading segments... Aug 7 21:09:47.214279799 (11h29m46.803624786s) [upload=11987]: Upload finished. err: <nil> Aug 7 21:10:37.572355395 (11h30m37.161700372s) [upload=11999]: Starting upload Aug 7 21:10:37.572410881 (11h30m37.161755868s) [upload=11999]: Done starting upload Aug 7 21:10:37.572467810 (11h30m37.161812797s) [upload=11999]: Uploading segments... Aug 7 21:10:37.572531982 (11h30m37.161876959s) [upload=11999]: Got next segment. Inline: true Aug 7 21:10:37.572536320 (11h30m37.161881307s) [upload=11999]: Waiting for error group managing segments... Aug 7 21:10:37.572570917 (11h30m37.161915904s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:37.572574955 (11h30m37.161919942s) [upload=11999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:37.572577520 (11h30m37.161922507s) [upload=11999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:37.838997375 (11h30m37.428342362s) [upload=11999]: Done uploading segments... Aug 7 21:10:37.839014699 (11h30m37.428359686s) [upload=11999]: Upload finished. err: <nil> Aug 7 21:11:31.258043368 (11h31m30.847388355s) [upload=12015]: Starting upload Aug 7 21:11:31.258101790 (11h31m30.847446777s) [upload=12015]: Done starting upload Aug 7 21:11:31.258163838 (11h31m30.847508825s) [upload=12015]: Uploading segments... Aug 7 21:11:31.258195730 (11h31m30.847540717s) [upload=12015]: Got next segment. Inline: true Aug 7 21:11:31.258199567 (11h31m30.847544565s) [upload=12015]: Waiting for error group managing segments... Aug 7 21:11:31.258240175 (11h31m30.847585162s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.258244853 (11h31m30.847589841s) [upload=12015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.258247358 (11h31m30.847592345s) [upload=12015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.535852595 (11h31m31.125197582s) [upload=12015]: Done uploading segments... Aug 7 21:11:31.535871041 (11h31m31.125216028s) [upload=12015]: Upload finished. err: <nil> Aug 7 21:11:57.912240700 (11h31m57.501585687s) [upload=12025]: Starting upload Aug 7 21:11:57.912288532 (11h31m57.501633519s) [upload=12025]: Done starting upload Aug 7 21:11:57.912345871 (11h31m57.501690858s) [upload=12025]: Uploading segments... Aug 7 21:11:57.912374877 (11h31m57.501719874s) [upload=12025]: Got next segment. Inline: true Aug 7 21:11:57.912378794 (11h31m57.501723781s) [upload=12025]: Waiting for error group managing segments... Aug 7 21:11:57.912425353 (11h31m57.501770401s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:57.912431234 (11h31m57.501776232s) [upload=12025]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:57.912435933 (11h31m57.50178095s) [upload=12025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:57.261749711 (11h32m56.851094698s) [upload=12025]: Done uploading segments... Aug 7 21:12:57.261780350 (11h32m56.851125337s) [upload=12025]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.979062305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.469555595s FAIL (2025-08-07 21:13:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 21:11:35.434526944 (11h31m35.023871921s) [upload=12017]: Done starting upload Aug 7 21:11:35.434621335 (11h31m35.023966362s) [upload=12017]: Uploading segments... Aug 7 21:11:35.434728259 (11h31m35.024073246s) [upload=12017]: Got next segment. Inline: false Aug 7 21:11:35.434762164 (11h31m35.024107151s) [upload=12017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:35.434766152 (11h31m35.024111139s) [upload=12017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:11:35.668275282 (11h31m35.257620259s) [seg_pos={0 0}, upload=12017]: Begin upload segment... Aug 7 21:11:35.668321981 (11h31m35.257666968s) [seg_pos={0 0}, upload=12017]: Done begin upload segment. Aug 7 21:11:35.668348001 (11h31m35.257692988s) [noise=true , address=45.91.116.7:4011 , piece_id=337XKW3GUCNSR6BLKFBPHU4IJUKJH6DYCBZW7ZXJ45SU6S2UZ2LA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.668411813 (11h31m35.2577568s) [noise=true , address=77.90.31.19:4843 , piece_id=AEUFO6KWQTAFBIQNKWDNQ3EL3XC2EKA7ULMHFYASCRS42YQMDLAA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.668705525 (11h31m35.258050512s) [noise=true , address=104.36.82.16:4294 , piece_id=JNKVRT2O7XLGKO5IHUIZCYAPKZ3SCABTR6FFS3M7GPSN7EBLQ4DQ, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.668877424 (11h31m35.258222401s) [noise=true , address=77.90.31.14:4662 , piece_id=X6U7XYH6AQHXARYULW2HSECW64RKAQNIREF3AAKJEK5COFZLLQ2A, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.668970081 (11h31m35.258315068s) [noise=true , address=77.90.31.10:4517 , piece_id=WPBIYBERJSHAGXUKBVSGJREJ6B37MNBOHNYR5GYRSX5N52G2RJ5A, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669186065 (11h31m35.258531052s) [noise=true , address=104.36.82.6:4876 , piece_id=OYGIZ34JUDMX64BFSPRTQIHKCW6ENRCQEYHE7QPNNS2BMPOR7M4Q, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669276127 (11h31m35.258621114s) [noise=true , address=45.91.116.14:4190 , piece_id=VRHZH224IVJRPT5P7C6S5EFPWVTUWZMEURBNSBPZUAONVVUERS3A, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669522218 (11h31m35.258867205s) [noise=true , address=45.91.116.8:4045 , piece_id=PX2WHDM4EYXXB3QZXUFDVEYNI23O5MF5N7UGUMLJY27DTI3L3LMA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669566662 (11h31m35.258911649s) [noise=true , address=77.90.31.16:4733 , piece_id=764U6SFKPDXTZ2LABZAIIMISMPJOBOE6TDZ734OWSFKSVFPVDBZQ, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669830978 (11h31m35.259175965s) [noise=true , address=45.91.116.10:4081 , piece_id=2P5ZU46HRSVSGTAM7R3G3B6PGYATVURICNCAM5R5YPKY44IFKX3Q, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.669869942 (11h31m35.259214929s) [noise=true , address=104.36.82.9:4005 , piece_id=LFBGDFRAYCSOBC6C47OHXPUIXSDEWL3HNAG2H4M25L3IHU3KGHJQ, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670054415 (11h31m35.259399393s) [noise=true , address=104.36.82.14:4223 , piece_id=4KR6G7VECDZRUMWFPYBXSK4IK27X6YXRSEGOPPLSFWRDTQEM66YA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670199142 (11h31m35.259544129s) [noise=true , address=45.91.116.6:4973 , piece_id=7MFZLVPY2BFFGMEIBBHTT6UUCFRYWQNDFRDEIN7WULORX37C2EWA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670281369 (11h31m35.259626357s) [noise=true , address=45.91.116.13:4153 , piece_id=JNIPJGHRPEJFGTZDBSOLPFPI6FNH23VWBO2O4DVFKOCUROKBAMQQ, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670499076 (11h31m35.259844053s) [noise=true , address=45.91.116.14:4191 , piece_id=64X5WB6NWW7QSOBVMO257IFCOLHCKIL6C3G7EMPVCR5PBMN2XPVA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:35.670765476 (11h31m35.260110453s) [noise=true , address=45.91.116.9:4262 , piece_id=R4TTQXN2BJKQ5WQ734B7WMDXZ3C4NAKBIQECI24WEZR3ZUSRPZEA, seg_pos={0 0}, upload=12017]: Uploading piece... Aug 7 21:11:37.305581962 (11h31m36.894926949s) [upload=12017]: Next returned nil segment Aug 7 21:11:37.305598983 (11h31m36.894943961s) [upload=12017]: Waiting for error group managing segments... Aug 7 21:11:37.704891137 (11h31m37.294236135s) [noise=true , address=45.91.116.10:4081 , piece_id=2P5ZU46HRSVSGTAM7R3G3B6PGYATVURICNCAM5R5YPKY44IFKX3Q, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.722340423 (11h31m37.31168541s) [noise=true , address=104.36.82.16:4294 , piece_id=JNKVRT2O7XLGKO5IHUIZCYAPKZ3SCABTR6FFS3M7GPSN7EBLQ4DQ, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.759508254 (11h31m37.348853241s) [noise=true , address=77.90.31.19:4843 , piece_id=AEUFO6KWQTAFBIQNKWDNQ3EL3XC2EKA7ULMHFYASCRS42YQMDLAA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.774868541 (11h31m37.364213528s) [noise=true , address=77.90.31.10:4517 , piece_id=WPBIYBERJSHAGXUKBVSGJREJ6B37MNBOHNYR5GYRSX5N52G2RJ5A, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.780756907 (11h31m37.370101894s) [noise=true , address=45.91.116.7:4011 , piece_id=337XKW3GUCNSR6BLKFBPHU4IJUKJH6DYCBZW7ZXJ45SU6S2UZ2LA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.816266847 (11h31m37.405611834s) [noise=true , address=104.36.82.6:4876 , piece_id=OYGIZ34JUDMX64BFSPRTQIHKCW6ENRCQEYHE7QPNNS2BMPOR7M4Q, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.828575126 (11h31m37.417920113s) [noise=true , address=77.90.31.16:4733 , piece_id=764U6SFKPDXTZ2LABZAIIMISMPJOBOE6TDZ734OWSFKSVFPVDBZQ, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.844647896 (11h31m37.433992883s) [noise=true , address=45.91.116.8:4045 , piece_id=PX2WHDM4EYXXB3QZXUFDVEYNI23O5MF5N7UGUMLJY27DTI3L3LMA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.867042233 (11h31m37.45638723s) [noise=true , address=104.36.82.14:4223 , piece_id=4KR6G7VECDZRUMWFPYBXSK4IK27X6YXRSEGOPPLSFWRDTQEM66YA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.888714938 (11h31m37.478059925s) [noise=true , address=45.91.116.6:4973 , piece_id=7MFZLVPY2BFFGMEIBBHTT6UUCFRYWQNDFRDEIN7WULORX37C2EWA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.963433603 (11h31m37.55277859s) [noise=true , address=45.91.116.13:4153 , piece_id=JNIPJGHRPEJFGTZDBSOLPFPI6FNH23VWBO2O4DVFKOCUROKBAMQQ, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:37.986126871 (11h31m37.575471858s) [noise=true , address=45.91.116.14:4191 , piece_id=64X5WB6NWW7QSOBVMO257IFCOLHCKIL6C3G7EMPVCR5PBMN2XPVA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:38.073190565 (11h31m37.662535552s) [noise=true , address=77.90.31.14:4662 , piece_id=X6U7XYH6AQHXARYULW2HSECW64RKAQNIREF3AAKJEK5COFZLLQ2A, seg_pos={0 0}, upload=12017]: Done uploading piece. err: <nil> Aug 7 21:11:38.073212426 (11h31m37.662557413s) [seg_pos={0 0}, upload=12017]: Segment reached optimal threshold of 13 pieces. Aug 7 21:11:38.073319682 (11h31m37.662664669s) [noise=true , address=45.91.116.9:4262 , piece_id=R4TTQXN2BJKQ5WQ734B7WMDXZ3C4NAKBIQECI24WEZR3ZUSRPZEA, seg_pos={0 0}, upload=12017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:38.073386219 (11h31m37.662731207s) [noise=true , address=45.91.116.14:4190 , piece_id=VRHZH224IVJRPT5P7C6S5EFPWVTUWZMEURBNSBPZUAONVVUERS3A, seg_pos={0 0}, upload=12017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:38.073506179 (11h31m37.662851166s) [noise=true , address=104.36.82.9:4005 , piece_id=LFBGDFRAYCSOBC6C47OHXPUIXSDEWL3HNAG2H4M25L3IHU3KGHJQ, seg_pos={0 0}, upload=12017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: write tcp 188.166.225.170:40678->104.36.82.9:4005: use of closed network connection; write tcp 188.166.225.170:40678->104.36.82.9:4005: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40678->104.36.82.9:4005: use of closed network connection Aug 7 21:11:38.073542368 (11h31m37.662887355s) [seg_pos={0 0}, upload=12017]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:11:38.073602473 (11h31m37.66294745s) [upload=12017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:11:38.073634404 (11h31m37.662979391s) [upload=12017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:38.377251317 (11h31m37.966596304s) [upload=12017]: Done uploading segments... Aug 7 21:11:38.377276456 (11h31m37.966621433s) [upload=12017]: Upload finished. err: <nil> Aug 7 21:12:29.590155214 (11h32m29.179500201s) [upload=12033]: Starting upload Aug 7 21:12:29.590208055 (11h32m29.179553042s) [upload=12033]: Done starting upload Aug 7 21:12:29.590306493 (11h32m29.17965148s) [upload=12033]: Uploading segments... Aug 7 21:12:29.590377449 (11h32m29.179722436s) [upload=12033]: Got next segment. Inline: false Aug 7 21:12:29.590398359 (11h32m29.179743346s) [upload=12033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:29.590401966 (11h32m29.179746953s) [upload=12033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:12:58.939555566 (11h32m58.528900553s) [upload=12033]: Next segment err: context deadline exceeded Aug 7 21:12:58.939613887 (11h32m58.528958874s) [upload=12033]: Done uploading segments... Aug 7 21:12:58.939626821 (11h32m58.528971798s) [upload=12033]: Upload finished. err: context deadline exceeded Duration: 6.469569852s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)