Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XXX.XXX.........................] duration: 5.700300128s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:11:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.700323763s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......X...] duration: 6.755280814s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:11:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.75529991s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.......XX....................XXXXXXXXXXXXXXX....X...........] duration: 3.780339256s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:11:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.780368301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XXX.XXX........................] duration: 5.911478653s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:11:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.911498931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......X..] duration: 6.719596812s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:10:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.719671093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......X.] duration: 14.315046178s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:10:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.3150687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [........XX....................XXXXXXXXXXXXXXX....X..........] duration: 3.887945185s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:10:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.887970656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......X] duration: 1m0.001420232s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:10:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001442875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX.XXX.......................] duration: 5.75892998s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:10:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.758955608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.........XX....................XXXXXXXXXXXXXXX....X.........] duration: 3.864189296s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:10:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.864215174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..........XX....................XXXXXXXXXXXXXXX....X........] duration: 3.798500044s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:09:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.798524109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXX.XXX......................] duration: 5.719251381s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:09:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.719262802s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...........XX....................XXXXXXXXXXXXXXX....X.......] duration: 3.854006395s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:09:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.854050108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XXX.XXX.....................] duration: 5.688167318s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:09:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.688192057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XXX.XXX....................] duration: 6.066947876s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:08:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.067029683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......] duration: 6.6930167s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:08:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.693034002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................XXX.XXX...................] duration: 6.644988506s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:08:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.645014797s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [............XX....................XXXXXXXXXXXXXXX....X......] duration: 3.656603285s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:08:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.656623774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................XXX.XXX..................] duration: 5.780506877s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:07:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.78052926s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX......] duration: 27.445615472s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:07:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.445645099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.............XX....................XXXXXXXXXXXXXXX....X.....] duration: 3.816402208s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:07:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.817227141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................XXX.XXX.................] duration: 6.653683027s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:07:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.653709426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XXX.XXX................] duration: 5.694470936s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:06:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.694493318s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.....] duration: 12.876394218s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.876423534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XXX.XXX...............] duration: 6.176008749s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:06:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.176032696s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..............XX....................XXXXXXXXXXXXXXX....X....] duration: 3.799275195s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:06:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.799299272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX....] duration: 27.705266147s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:05:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.705292436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XXX.XXX..............] duration: 6.304097472s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:05:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.304122741s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...............XX....................XXXXXXXXXXXXXXX....X...] duration: 3.692248491s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:05:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.692272315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX...] duration: 27.972951823s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:04:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.973321869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XXX.XXX.............] duration: 5.687144829s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:04:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.687170158s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [................XX....................XXXXXXXXXXXXXXX....X..] duration: 3.737732043s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:04:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.737753955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.................XX....................XXXXXXXXXXXXXXX....X.] duration: 3.802071718s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:04:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.802095452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX..] duration: 6.641429132s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:03:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.641451746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XXX.XXX............] duration: 5.687890508s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:03:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.687909574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..................XX....................XXXXXXXXXXXXXXX....X] duration: 5.001313852s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:03:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001334841s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.] duration: 6.872501802s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:03:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.872515039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX] duration: 1m0.001086773s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:03:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001097975s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...................XX....................XXXXXXXXXXXXXXX....] duration: 3.76752199s trace: https://jaeger.dev.storj.io/trace/6bb06f369a5d2b5b FAIL (2026-05-14 00:02:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.767545255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XXX.XXX...........] duration: 5.788336015s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:02:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.788358547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [..XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...X] duration: 1m0.001017625s trace: https://jaeger.dev.storj.io/trace/1b42330705019b19 FAIL (2026-05-14 00:02:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001037794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [....................XX....................XXXXXXXXXXXXXXX...] duration: 3.853929785s trace: https://jaeger.dev.storj.io/trace/6bb06f369a5d2b5b FAIL (2026-05-14 00:01:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.853952358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................XXX.XXX..........] duration: 5.734631249s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:01:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.734654504s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX...] duration: 6.89373874s trace: https://jaeger.dev.storj.io/trace/7c340024ca11ce85 FAIL (2026-05-14 00:00:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.893762867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................XXX.XXX.........] duration: 5.798792128s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:00:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.798816412s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.....................XX....................XXXXXXXXXXXXXXX..] duration: 3.719710786s trace: https://jaeger.dev.storj.io/trace/6bb06f369a5d2b5b FAIL (2026-05-14 00:00:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.719746594s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX..] duration: 7.097944937s trace: https://jaeger.dev.storj.io/trace/7c340024ca11ce85 FAIL (2026-05-14 00:00:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.097958691s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.....XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX.] duration: 13.525289397s trace: https://jaeger.dev.storj.io/trace/7c340024ca11ce85 FAIL (2026-05-14 00:00:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.525302602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XXX.XXX........] duration: 8.09205545s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:59:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 8.092083462s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [......XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX] duration: 1m0.0005237s trace: https://jaeger.dev.storj.io/trace/7c340024ca11ce85 FAIL (2026-05-13 23:59:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000536174s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)