Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXX....X.......................................] duration: 3.686205849s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:32:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.686231467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXX.XXX..................................................] duration: 6.261134521s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:32:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.261148007s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X..XXX.X.X.XXX...XX.......X...................X.......X..X.] duration: 41.081699767s trace: https://jaeger.dev.storj.io/trace/667ea420927a5c4e FAIL (2026-05-14 00:32:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.081714465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X..XXX.X.X.XXX...XX.......X...................X.......X..X] duration: 1m0.000684947s trace: https://jaeger.dev.storj.io/trace/667ea420927a5c4e FAIL (2026-05-14 00:31:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000725074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXX....X......................................] duration: 3.644469809s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:31:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.644493814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XXX.XXX.................................................] duration: 5.815166541s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:31:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.815195885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XXX.XXX................................................] duration: 6.263620857s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:30:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.263645062s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXX....X.....................................] duration: 3.769175469s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:30:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.769232098s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...X..XXX.X.X.XXX...XX.......X...................X.......X..] duration: 20.717959697s trace: https://jaeger.dev.storj.io/trace/1bd1dbed059321bc FAIL (2026-05-14 00:30:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.717987137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XXX.XXX...............................................] duration: 5.854140441s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.854167443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XXX.XXX..............................................] duration: 6.228941643s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:29:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.228958855s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXX....X....................................] duration: 3.738197927s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:29:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.73821412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [....X..XXX.X.X.XXX...XX.......X...................X.......X.] duration: 6.790197332s trace: https://jaeger.dev.storj.io/trace/1bd1dbed059321bc FAIL (2026-05-14 00:29:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.790210688s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.....X..XXX.X.X.XXX...XX.......X...................X.......X] duration: 1m0.000829452s trace: https://jaeger.dev.storj.io/trace/1bd1dbed059321bc FAIL (2026-05-14 00:29:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000848409s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXX....X...................................] duration: 3.772093858s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:28:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.772117263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XXX.XXX.............................................] duration: 5.810415705s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.810442966s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X.....X..XXX.X.X.XXX...XX.......X...................X.......] duration: 16.589705398s trace: https://jaeger.dev.storj.io/trace/5129f1b7f5b046d8 FAIL (2026-05-14 00:27:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.58972677s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXX....X..................................] duration: 3.560184356s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:27:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.560616261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XXX.XXX............................................] duration: 6.560819459s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:27:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.560843656s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X.....X..XXX.X.X.XXX...XX.......X...................X......] duration: 17.929093453s trace: https://jaeger.dev.storj.io/trace/5129f1b7f5b046d8 FAIL (2026-05-14 00:27:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.929105596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXX....X.................................] duration: 3.732636356s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:27:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.732662776s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..X.....X..XXX.X.X.XXX...XX.......X...................X.....] duration: 21.280833919s trace: https://jaeger.dev.storj.io/trace/5129f1b7f5b046d8 FAIL (2026-05-14 00:27:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.280849799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXX....X................................] duration: 3.763608798s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:26:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.763634427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XXX.XXX...........................................] duration: 5.69898169s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:26:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.699016948s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X..X.....X..XXX.X.X.XXX...XX.......X...................X....] duration: 6.592636548s trace: https://jaeger.dev.storj.io/trace/5129f1b7f5b046d8 FAIL (2026-05-14 00:25:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.592652438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXX....X...............................] duration: 3.797174124s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:25:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.797207648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XXX.XXX..........................................] duration: 5.742299139s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:25:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.742319838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X..X.....X..XXX.X.X.XXX...XX.......X...................X...] duration: 9.577415355s trace: https://jaeger.dev.storj.io/trace/5129f1b7f5b046d8 FAIL (2026-05-14 00:25:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.577552468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..........XXXXXXXXXXXXXXX....X..............................] duration: 3.583321166s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:25:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.583342216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XXX.XXX.........................................] duration: 6.249860137s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.249883192s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X..X.....X..XXX.X.X.XXX...XX.......X...................X..] duration: 16.999705568s trace: https://jaeger.dev.storj.io/trace/5129f1b7f5b046d8 FAIL (2026-05-14 00:24:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.999717821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XXX.XXX........................................] duration: 5.799410106s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:24:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.799437409s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...........XXXXXXXXXXXXXXX....X.............................] duration: 3.739405162s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:24:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.739426504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...X..X.....X..XXX.X.X.XXX...XX.......X...................X.] duration: 22.15131602s trace: https://jaeger.dev.storj.io/trace/5129f1b7f5b046d8 FAIL (2026-05-14 00:24:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.151327692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XXX.XXX.......................................] duration: 5.794381441s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:24:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.79440757s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [....X..X.....X..XXX.X.X.XXX...XX.......X...................X] duration: 1m0.001569127s trace: https://jaeger.dev.storj.io/trace/5129f1b7f5b046d8 FAIL (2026-05-14 00:23:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.0015969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [............XXXXXXXXXXXXXXX....X............................] duration: 3.840635408s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:23:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.840656359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XXX.XXX......................................] duration: 6.699002071s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:22:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.699032169s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.............XXXXXXXXXXXXXXX....X...........................] duration: 3.720762673s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:22:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.720786057s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.....X..X.....X..XXX.X.X.XXX...XX.......X...................] duration: 6.438295206s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:22:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.438324231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..............XXXXXXXXXXXXXXX....X..........................] duration: 3.657488455s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:22:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.65750729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................XXX.XXX.....................................] duration: 6.328571252s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:22:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 6.328589868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [......X..X.....X..XXX.X.X.XXX...XX.......X..................] duration: 7.140949416s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:21:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.14099885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...............XXXXXXXXXXXXXXX....X.........................] duration: 3.660443112s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:21:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.660467881s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.......X..X.....X..XXX.X.X.XXX...XX.......X.................] duration: 56.724847377s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:21:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.724872845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [................XXXXXXXXXXXXXXX....X........................] duration: 3.826919198s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:21:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.826944245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................XXX.XXX....................................] duration: 5.754972835s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:21:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.755000138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................XXX.XXX...................................] duration: 5.684445154s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-14 00:20:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 5.684471094s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.................XXXXXXXXXXXXXXX....X.......................] duration: 3.731964836s trace: https://jaeger.dev.storj.io/trace/2786f877eb009a88 FAIL (2026-05-14 00:20:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.731988502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [........X..X.....X..XXX.X.X.XXX...XX.......X................] duration: 41.041851553s trace: https://jaeger.dev.storj.io/trace/4d2498814b0ef33 FAIL (2026-05-14 00:20:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.041873936s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)