Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.................XXXXXXXXXXXXXXX....X.......................] duration: 3.731964836s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:20:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.731988502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [........X..X.....X..XXX.X.X.XXX...XX.......X................] duration: 41.041851553s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:20:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.041873936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..................XXXXXXXXXXXXXXX....X......................] duration: 3.696699453s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:19:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.696726796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.........X..X.....X..XXX.X.X.XXX...XX.......X...............] duration: 19.482965242s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:19:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.482986372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XXX.XXX..................................] duration: 5.75405913s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:19:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.754085321s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...................XXXXXXXXXXXXXXX....X.....................] duration: 3.744242034s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:18:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.744257354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..........X..X.....X..XXX.X.X.XXX...XX.......X..............] duration: 7.609385999s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:18:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.609409875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XXX.XXX.................................] duration: 5.757967938s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:18:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.75799554s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [....................XXXXXXXXXXXXXXX....X....................] duration: 3.740479104s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:17:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.740506427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XXX.XXX................................] duration: 6.777686126s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:17:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.77769912s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...........X..X.....X..XXX.X.X.XXX...XX.......X.............] duration: 6.615539018s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:17:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.615564707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X....................XXXXXXXXXXXXXXX....X...................] duration: 3.59170726s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:17:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.591729262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [............X..X.....X..XXX.X.X.XXX...XX.......X............] duration: 7.569643714s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:16:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.569680625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XX....................XXXXXXXXXXXXXXX....X..................] duration: 3.671792729s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:16:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.671819319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XXX.XXX...............................] duration: 5.835400591s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:16:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.83543099s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.............X..X.....X..XXX.X.X.XXX...XX.......X...........] duration: 6.455921593s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:16:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.455944698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XXX.XXX..............................] duration: 5.73005367s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:15:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.730075532s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X.............X..X.....X..XXX.X.X.XXX...XX.......X..........] duration: 6.541803473s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:15:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.541834914s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.XX....................XXXXXXXXXXXXXXX....X.................] duration: 3.68530594s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:15:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.685328341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XXX.XXX.............................] duration: 5.837273408s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:15:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.837295871s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X.............X..X.....X..XXX.X.X.XXX...XX.......X.........] duration: 6.656533946s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:15:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.656561109s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..XX....................XXXXXXXXXXXXXXX....X................] duration: 3.655516241s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:15:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.655528463s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X.............X..X.....X..XXX.X.X.XXX...XX.......X........] duration: 6.634020936s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:14:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.634052074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...X.............X..X.....X..XXX.X.X.XXX...XX.......X.......] duration: 6.66572896s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:14:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.666015666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XXX.XXX............................] duration: 6.856076675s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:14:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.856106671s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...XX....................XXXXXXXXXXXXXXX....X...............] duration: 3.747709487s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:14:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.747727541s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [....X.............X..X.....X..XXX.X.X.XXX...XX.......X......] duration: 6.414461833s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:13:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.414483905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [....XX....................XXXXXXXXXXXXXXX....X..............] duration: 3.634554862s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:13:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.634583687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.....XX....................XXXXXXXXXXXXXXX....X.............] duration: 3.671926798s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:13:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.671952758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.....X.............X..X.....X..XXX.X.X.XXX...XX.......X.....] duration: 24.483301959s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:13:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.483344109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XXX.XXX...........................] duration: 5.72690801s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:13:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.726931736s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [......XX....................XXXXXXXXXXXXXXX....X............] duration: 4.102192217s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:12:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.102217316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [X.....X.............X..X.....X..XXX.X.X.XXX...XX.......X....] duration: 6.512291047s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:12:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.512312889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................XXX.XXX..........................] duration: 5.852215295s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:12:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.852239371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XXX.XXX.........................] duration: 5.700300128s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:11:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.700323763s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......X...] duration: 6.755280814s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:11:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.75529991s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.......XX....................XXXXXXXXXXXXXXX....X...........] duration: 3.780339256s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:11:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.780368301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XXX.XXX........................] duration: 5.911478653s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:11:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.911498931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......X..] duration: 6.719596812s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:10:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.719671093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......X.] duration: 14.315046178s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:10:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.3150687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [........XX....................XXXXXXXXXXXXXXX....X..........] duration: 3.887945185s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:10:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.887970656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......X] duration: 1m0.001420232s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:10:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001442875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX.XXX.......................] duration: 5.75892998s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:10:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.758955608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.........XX....................XXXXXXXXXXXXXXX....X.........] duration: 3.864189296s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:10:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.864215174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..........XX....................XXXXXXXXXXXXXXX....X........] duration: 3.798500044s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:09:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.798524109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXX.XXX......................] duration: 5.719251381s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:09:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.719262802s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...........XX....................XXXXXXXXXXXXXXX....X.......] duration: 3.854006395s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:09:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.854050108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XXX.XXX.....................] duration: 5.688167318s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:09:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.688192057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XXX.XXX....................] duration: 6.066947876s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:08:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.067029683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..X.XX.....X.............X..X.....X..XXX.X.X.XXX...XX.......] duration: 6.6930167s trace: https://jaeger.dev.storj.io/trace/1dfe8e1cb2978f3f FAIL (2026-05-14 00:08:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.693034002s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)