Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.....................XX....................XXXXXXXXXXXXXXX..] duration: 3.719710786s trace: https://jaeger.dev.storj.io/trace/6bb06f369a5d2b5b FAIL (2026-05-14 00:00:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.719746594s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX..] duration: 7.097944937s trace: https://jaeger.dev.storj.io/trace/7c340024ca11ce85 FAIL (2026-05-14 00:00:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.097958691s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.....XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX.] duration: 13.525289397s trace: https://jaeger.dev.storj.io/trace/7c340024ca11ce85 FAIL (2026-05-14 00:00:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.525302602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XXX.XXX........] duration: 8.09205545s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:59:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 8.092083462s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [......XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XXX] duration: 1m0.0005237s trace: https://jaeger.dev.storj.io/trace/7c340024ca11ce85 FAIL (2026-05-13 23:59:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000536174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [......................XX....................XXXXXXXXXXXXXXX.] duration: 5.000423049s trace: https://jaeger.dev.storj.io/trace/6bb06f369a5d2b5b FAIL (2026-05-13 23:59:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000454217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XXX.XXX.......] duration: 18.157449438s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:59:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 18.157473192s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.......................XX....................XXXXXXXXXXXXXXX] duration: 5.001005523s trace: https://jaeger.dev.storj.io/trace/6bb06f369a5d2b5b FAIL (2026-05-13 23:59:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001017436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.......XX...X..X.XX.....X.............X..X.....X..XXX.X.X.XX] duration: 1m0.000840823s trace: https://jaeger.dev.storj.io/trace/4a4a3e62d786da0b FAIL (2026-05-13 23:58:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00084995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................XXX.XXX......] duration: 17.423113853s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:58:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 17.423137788s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [........................XX....................XXXXXXXXXXXXXX] duration: 5.001414419s trace: https://jaeger.dev.storj.io/trace/3e1d90d27e51a0eb FAIL (2026-05-13 23:58:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001437372s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [........XX...X..X.XX.....X.............X..X.....X..XXX.X.X.X] duration: 1m0.000439801s trace: https://jaeger.dev.storj.io/trace/78b7503839be96f0 FAIL (2026-05-13 23:57:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000457894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XXX.XXX.....] duration: 23.868817126s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:57:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 23.86884551s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.........................XX....................XXXXXXXXXXXXX] duration: 5.001747357s trace: https://jaeger.dev.storj.io/trace/716fac4a86ee5394 FAIL (2026-05-13 23:57:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001767045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................XXX.XXX....] duration: 17.602097503s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:56:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 17.602110299s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..........................XX....................XXXXXXXXXXXX] duration: 5.001654153s trace: https://jaeger.dev.storj.io/trace/a93c0847dfbcfbf FAIL (2026-05-13 23:56:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001677457s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.........XX...X..X.XX.....X.............X..X.....X..XXX.X.X.] duration: 17.90060017s trace: https://jaeger.dev.storj.io/trace/79748dca761ca368 FAIL (2026-05-13 23:56:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.900611853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XXX.XXX...] duration: 28.974822997s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:56:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 28.974857091s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..........XX...X..X.XX.....X.............X..X.....X..XXX.X.X] duration: 1m0.000546157s trace: https://jaeger.dev.storj.io/trace/79748dca761ca368 FAIL (2026-05-13 23:56:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000563119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................................XXX.XXX..] duration: 17.048042363s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:56:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 17.048066879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...........................XX....................XXXXXXXXXXX] duration: 5.001511801s trace: https://jaeger.dev.storj.io/trace/16e91bc173758e73 FAIL (2026-05-13 23:55:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001522943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [............................XX....................XXXXXXXXXX] duration: 5.000966918s trace: https://jaeger.dev.storj.io/trace/4847263dd49d270a FAIL (2026-05-13 23:55:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001087729s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.............................XX....................XXXXXXXXX] duration: 5.002040668s trace: https://jaeger.dev.storj.io/trace/1888f2365ea168c5 FAIL (2026-05-13 23:55:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002059944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................................XXX.XXX.] duration: 24.432512487s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 24.432531834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..............................XX....................XXXXXXXX] duration: 5.000721331s trace: https://jaeger.dev.storj.io/trace/5ec170fec5fd0e90 FAIL (2026-05-13 23:54:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000771799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...........XX...X..X.XX.....X.............X..X.....X..XXX.X.] duration: 30.405277074s trace: https://jaeger.dev.storj.io/trace/5aa44147e07aee20 FAIL (2026-05-13 23:54:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.405327519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXX.XXX] duration: 30.00191625s trace: https://jaeger.dev.storj.io/trace/1cb3ad1e3120325 FAIL (2026-05-13 23:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA) context deadline exceeded ======LOGS====== May 13 23:54:09.101041394 (317h6m19.621865139s) [upload=303759]: Starting upload May 13 23:54:09.101150321 (317h6m19.621974066s) [upload=303759]: Done starting upload May 13 23:54:09.101523945 (317h6m19.62234769s) [upload=303759]: Uploading segments... May 13 23:54:09.101671446 (317h6m19.622495191s) [upload=303759]: Got next segment. Inline: false May 13 23:54:09.101764053 (317h6m19.622587788s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:54:09.101813186 (317h6m19.622636931s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:54:09.342652473 (317h6m19.863476219s) [seg_pos={0 0}, upload=303759]: Begin upload segment... May 13 23:54:09.342708981 (317h6m19.863532726s) [seg_pos={0 0}, upload=303759]: Done begin upload segment. May 13 23:54:09.342734078 (317h6m19.863557824s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.342806718 (317h6m19.863630463s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343405821 (317h6m19.864229556s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.343736492 (317h6m19.864560257s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344273326 (317h6m19.865097072s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344684090 (317h6m19.865507835s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.344692727 (317h6m19.865516472s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345011675 (317h6m19.86583542s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345015542 (317h6m19.865839287s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345336104 (317h6m19.866159849s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345446635 (317h6m19.86627038s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345625066 (317h6m19.866448811s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.345928595 (317h6m19.86675234s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346196556 (317h6m19.867020301s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346515965 (317h6m19.86733971s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:09.346596399 (317h6m19.867420134s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Uploading piece... May 13 23:54:11.479134008 (317h6m21.999957743s) [upload=303759]: Next returned nil segment May 13 23:54:11.479176098 (317h6m21.999999833s) [upload=303759]: Waiting for error group managing segments... May 13 23:54:11.875349259 (317h6m22.396173004s) [noise=true , address=45.91.116.5:4299 , piece_id=Z3I66FUKFYFVRK5QISCU3VO3WSQGADV7VI45HWPKKCC4Y7R5JJGQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:11.929014675 (317h6m22.44983842s) [noise=true , address=45.91.116.4:4940 , piece_id=4FP6TLZHKHWVV5YYI44P7X4B23X3MBYP4HWDGAPN4P723FVO7FMA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.522398786 (317h6m27.043222531s) [noise=true , address=77.90.31.11:4555 , piece_id=2TQNISXX4GF43FVAZBQLXIDV3VIQFQYWVNSWNRNBVF4RHXQGWB3A, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.677455972 (317h6m27.198279717s) [noise=true , address=145.79.227.12:4770 , piece_id=IML2LWFBEHA7QOC4H7WZYZ4PMEK5RSTHFPRGPGY7N2HLKIXGJWJQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.904600146 (317h6m27.425423891s) [noise=true , address=145.79.227.42:4124 , piece_id=T5F3Z57TZ5CRE2PAKUEF5BTQH2VENBAZIOUEVHR23HRO5RPBATZA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:16.962748155 (317h6m27.4835719s) [noise=true , address=145.79.227.13:4804 , piece_id=CK6FOK2W7YIUAE3YJ6XAY3NR6FIQZ7BYKTBEELIFHBASPV3IK6GA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:19.824741116 (317h6m30.345564861s) [noise=true , address=145.79.227.29:4941 , piece_id=I6NEQI2B72T2XC37GD5XX2UUUISF6K6ZMBQPNKY3WZ73ZEM7F7LA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:23.318972131 (317h6m33.839795876s) [noise=true , address=145.79.227.20:4437 , piece_id=6BTVPMO5EYNPTLX5CJUQLETFI4W3WNVKLOUTOQZNOFCFMCEAC2FQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:28.699626931 (317h6m39.220450676s) [noise=true , address=145.79.227.45:4234 , piece_id=WQINO2KZFCOKCUR2LKD63QGYU4U7QXO7DZ5Y4K65ZIE6NGRHTHSA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:30.580138066 (317h6m41.100961811s) [noise=true , address=145.79.227.4:4188 , piece_id=LSJSOJIL7GFMQRRVSYBSCYZ34MJK4K3M2XRBHTTXUAOPGRNKJTRA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:31.614065620 (317h6m42.134889355s) [noise=true , address=77.90.31.13:4626 , piece_id=ZAZIJR7MONOZTYDGRB45EJHI4AT4AO2HICUFTNEQ5M4NK5RJSEYA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:34.869340665 (317h6m45.39016441s) [noise=true , address=145.79.227.2:4117 , piece_id=B7N34ENGOW346JAOQ4TP2R7UURBRRCNSI2VJK3CJDDDSF7ZUBVCA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539227446 (317h6m46.060051201s) [noise=true , address=77.90.31.21:4879 , piece_id=W3LN7Y3TQ6QZHFMFQ3I47EYX5O4EUEPGQE6VCTHCRB3QJCSVGCFQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: <nil> May 13 23:54:35.539249819 (317h6m46.060073574s) [seg_pos={0 0}, upload=303759]: Satisfied cohort requirements after 13 pieces May 13 23:54:35.539509113 (317h6m46.060332848s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539517068 (317h6m46.060340813s) [noise=true , address=145.79.227.48:4306 , piece_id=GOM6KEXBA3I6VU3NYSG3JBEFIO5YWGEWM6STZOLCQQONXBEFORKA, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539532217 (317h6m46.060355962s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52152->145.79.227.15:4878: use of closed network connection May 13 23:54:35.539539841 (317h6m46.060363587s) [noise=true , address=145.79.227.15:4878 , piece_id=GOZMXBY3FYWFDQ6X45JGBWHKJT36LFBUVS4XJIOQTVGBO3ZI2DWQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539679248 (317h6m46.060502993s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF May 13 23:54:35.539685650 (317h6m46.060509395s) [noise=true , address=145.79.227.44:4197 , piece_id=YGQORIGW76DRF4XFAKRYP64242S7EM43ED73OLUQ2WMGLPWKRMXQ, seg_pos={0 0}, upload=303759]: Long tail cancellation due to optimal threshold May 13 23:54:35.539760081 (317h6m46.060583826s) [seg_pos={0 0}, upload=303759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:54:35.539831719 (317h6m46.060655464s) [upload=303759]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:54:35.539836608 (317h6m46.060660353s) [upload=303759]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:54:35.834703623 (317h6m46.355527358s) [upload=303759]: Done uploading segments... May 13 23:54:35.834721307 (317h6m46.355545042s) [upload=303759]: Upload finished. err: <nil> Duration: 30.001934094s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X...........XX...X..X.XX.....X.............X..X.....X..XXX.X] duration: 1m0.00119632s trace: https://jaeger.dev.storj.io/trace/5aa44147e07aee20 FAIL (2026-05-13 23:54:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001219444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...............................XX....................XXXXXXX] duration: 5.001065731s trace: https://jaeger.dev.storj.io/trace/13a7ab1e274f3bf4 FAIL (2026-05-13 23:54:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001086522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXX.XX] duration: 30.002398757s trace: https://jaeger.dev.storj.io/trace/5cf4820d679cde97 FAIL (2026-05-13 23:53:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, Piece ID: U5JRDUWIYO4AX55J4NTWUVQTCYHRAC2H6ZXKMVP5NI6P6UCK5NMQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WLGYM3NPY5UF467SY72JH27BOFUBVXE46BVHHNP472LQDSGKODOQ) context deadline exceeded; piecestore: (Node ID: 12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, Piece ID: FPIDRLUJAIR4TXFOCQZ2L74CT5OLQOQSPTNZ3YK3M6LEBM2P5UMQ) context deadline exceeded; piecestore: (Node ID: 129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy, Piece ID: KJCSABS7PG7WDVGZDIV4EQ24JPDJXTKOXKRBC4GPIILBNZYXROXA) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: P4U7DRJHXAAWRIFDCTHT33JSTKJPA2E6LPRB7DMYNDBHXNO5IR2Q) context deadline exceeded; piecestore: (Node ID: 124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, Piece ID: AHZYXA5XCLFEMCWEQ5NQ4MUAREZNWQ7JNZEF6Y7UGKZXXKN2MHMA) context deadline exceeded ======LOGS====== May 13 23:53:03.919169138 (317h5m14.439992883s) [upload=303742]: Starting upload May 13 23:53:03.919218762 (317h5m14.440042517s) [upload=303742]: Done starting upload May 13 23:53:03.919348851 (317h5m14.440172596s) [upload=303742]: Uploading segments... May 13 23:53:03.919443261 (317h5m14.440267006s) [upload=303742]: Got next segment. Inline: false May 13 23:53:03.919479410 (317h5m14.440303155s) [upload=303742]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:53:03.919483918 (317h5m14.440307663s) [upload=303742]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:53:04.155862374 (317h5m14.676686129s) [seg_pos={0 0}, upload=303742]: Begin upload segment... May 13 23:53:04.155936665 (317h5m14.67676043s) [seg_pos={0 0}, upload=303742]: Done begin upload segment. May 13 23:53:04.155979788 (317h5m14.676803533s) [noise=true , address=145.79.227.50:4377 , piece_id=4N5SSNKSFKRKZMRQTWTVQZHWBKX7H2PNBQJ3VYFBXX544UXBY37A, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.155986541 (317h5m14.676810276s) [noise=true , address=145.79.227.33:4873 , piece_id=Q2XQ2SKSNVSYX3EAN63MDELHAW24C5W2AMCTX2U4WZWX7T7O6HOQ, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.156109285 (317h5m14.67693303s) [noise=true , address=145.79.227.39:4018 , piece_id=UPTS6SD4ONOFCUDYOK4XLDEXTSYDDAUTHPEAZEV6MD356JJ2HUBA, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.156775877 (317h5m14.677599622s) [noise=true , address=145.79.227.6:4262 , piece_id=A6QNPJIZ2M4ARXOIPYJY67YYOPLMX3ABEF46TETJVGNT3YIXHGHQ, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.157327980 (317h5m14.678151735s) [noise=true , address=145.79.227.46:4604 , piece_id=KJCSABS7PG7WDVGZDIV4EQ24JPDJXTKOXKRBC4GPIILBNZYXROXA, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.157768892 (317h5m14.678592637s) [noise=true , address=145.79.227.42:4122 , piece_id=AHZYXA5XCLFEMCWEQ5NQ4MUAREZNWQ7JNZEF6Y7UGKZXXKN2MHMA, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.158221275 (317h5m14.67904502s) [noise=true , address=145.79.227.51:4410 , piece_id=YQF3IOBA7ZRBHXKZUHAYJEK5XKDTZGDT53LE4TLB52JRYTPPHKSA, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.158611559 (317h5m14.679435305s) [noise=true , address=45.91.116.5:4299 , piece_id=PT5PFSS7B6VVPACPLPKS7AGO2DIEWO2QDMM574FEHWJQYSMD3FWA, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.159053233 (317h5m14.679876978s) [noise=true , address=145.79.227.44:4196 , piece_id=VIDIZ3VRM3LFB7BJIZJ4KTEUZACUWMHSUKPX2MBY3GB3AKN56GLQ, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.159279324 (317h5m14.680103069s) [noise=true , address=145.79.227.13:4804 , piece_id=FPIDRLUJAIR4TXFOCQZ2L74CT5OLQOQSPTNZ3YK3M6LEBM2P5UMQ, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.159771734 (317h5m14.680595479s) [noise=true , address=145.79.227.36:4517 , piece_id=P4U7DRJHXAAWRIFDCTHT33JSTKJPA2E6LPRB7DMYNDBHXNO5IR2Q, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.160212705 (317h5m14.68103645s) [noise=true , address=145.79.227.12:4769 , piece_id=U5JRDUWIYO4AX55J4NTWUVQTCYHRAC2H6ZXKMVP5NI6P6UCK5NMQ, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.160652224 (317h5m14.681475969s) [noise=true , address=145.79.227.14:4842 , piece_id=VRLMNNPQFAGMKG3NDDCKZQG7XQ6LWYPX6Z47XAO6UNBOLFOXSNBA, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.160835644 (317h5m14.681659389s) [noise=true , address=77.90.31.11:4556 , piece_id=WLGYM3NPY5UF467SY72JH27BOFUBVXE46BVHHNP472LQDSGKODOQ, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.161269050 (317h5m14.682092795s) [noise=true , address=77.90.31.15:4701 , piece_id=JJ65CNC4T5KDV5MRYDXMUCK5NFVJMREOTM7BR2LAJ3MW7YMDVQ6A, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:04.161676789 (317h5m14.682500534s) [noise=true , address=45.91.116.9:4262 , piece_id=SNQGVY7PF3X7EDRPKS4CUBRSR5L3BXWJMQLHYC5YE5RDUJRBEWAA, seg_pos={0 0}, upload=303742]: Uploading piece... May 13 23:53:06.375463971 (317h5m16.896287716s) [upload=303742]: Next returned nil segment May 13 23:53:06.375479661 (317h5m16.896303406s) [upload=303742]: Waiting for error group managing segments... May 13 23:53:06.805732974 (317h5m17.326556719s) [noise=true , address=45.91.116.5:4299 , piece_id=PT5PFSS7B6VVPACPLPKS7AGO2DIEWO2QDMM574FEHWJQYSMD3FWA, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:06.822526180 (317h5m17.343349925s) [noise=true , address=45.91.116.9:4262 , piece_id=SNQGVY7PF3X7EDRPKS4CUBRSR5L3BXWJMQLHYC5YE5RDUJRBEWAA, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:11.118942612 (317h5m21.639766357s) [noise=true , address=77.90.31.11:4556 , piece_id=WLGYM3NPY5UF467SY72JH27BOFUBVXE46BVHHNP472LQDSGKODOQ, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:15.235288188 (317h5m25.756111933s) [noise=true , address=145.79.227.6:4262 , piece_id=A6QNPJIZ2M4ARXOIPYJY67YYOPLMX3ABEF46TETJVGNT3YIXHGHQ, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:15.340244899 (317h5m25.861068664s) [noise=true , address=145.79.227.14:4842 , piece_id=VRLMNNPQFAGMKG3NDDCKZQG7XQ6LWYPX6Z47XAO6UNBOLFOXSNBA, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:16.635635467 (317h5m27.156459202s) [noise=true , address=145.79.227.42:4122 , piece_id=AHZYXA5XCLFEMCWEQ5NQ4MUAREZNWQ7JNZEF6Y7UGKZXXKN2MHMA, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:17.677424082 (317h5m28.198247827s) [noise=true , address=145.79.227.46:4604 , piece_id=KJCSABS7PG7WDVGZDIV4EQ24JPDJXTKOXKRBC4GPIILBNZYXROXA, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:17.743795434 (317h5m28.264619179s) [noise=true , address=145.79.227.33:4873 , piece_id=Q2XQ2SKSNVSYX3EAN63MDELHAW24C5W2AMCTX2U4WZWX7T7O6HOQ, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:23.148786665 (317h5m33.669610401s) [noise=true , address=145.79.227.13:4804 , piece_id=FPIDRLUJAIR4TXFOCQZ2L74CT5OLQOQSPTNZ3YK3M6LEBM2P5UMQ, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:24.658664054 (317h5m35.179487799s) [noise=true , address=145.79.227.36:4517 , piece_id=P4U7DRJHXAAWRIFDCTHT33JSTKJPA2E6LPRB7DMYNDBHXNO5IR2Q, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:25.167406844 (317h5m35.688230589s) [noise=true , address=145.79.227.44:4196 , piece_id=VIDIZ3VRM3LFB7BJIZJ4KTEUZACUWMHSUKPX2MBY3GB3AKN56GLQ, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:26.701685896 (317h5m37.222509642s) [noise=true , address=145.79.227.12:4769 , piece_id=U5JRDUWIYO4AX55J4NTWUVQTCYHRAC2H6ZXKMVP5NI6P6UCK5NMQ, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:30.967634663 (317h5m41.488458408s) [noise=true , address=145.79.227.39:4018 , piece_id=UPTS6SD4ONOFCUDYOK4XLDEXTSYDDAUTHPEAZEV6MD356JJ2HUBA, seg_pos={0 0}, upload=303742]: Done uploading piece. err: <nil> May 13 23:53:30.967653729 (317h5m41.488477474s) [seg_pos={0 0}, upload=303742]: Satisfied cohort requirements after 13 pieces May 13 23:53:30.967794437 (317h5m41.488618182s) [noise=true , address=145.79.227.50:4377 , piece_id=4N5SSNKSFKRKZMRQTWTVQZHWBKX7H2PNBQJ3VYFBXX544UXBY37A, seg_pos={0 0}, upload=303742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126HKpC7sciWFTxNaH5Hjs6BBuh8NRgu8fA4cu4qcK29zfJWg7R): protocol: expected piece hash; context canceled; EOF May 13 23:53:30.967799277 (317h5m41.488623022s) [noise=true , address=145.79.227.50:4377 , piece_id=4N5SSNKSFKRKZMRQTWTVQZHWBKX7H2PNBQJ3VYFBXX544UXBY37A, seg_pos={0 0}, upload=303742]: Long tail cancellation due to optimal threshold May 13 23:53:30.967819475 (317h5m41.48864322s) [noise=true , address=145.79.227.51:4410 , piece_id=YQF3IOBA7ZRBHXKZUHAYJEK5XKDTZGDT53LE4TLB52JRYTPPHKSA, seg_pos={0 0}, upload=303742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF May 13 23:53:30.967826438 (317h5m41.488650173s) [noise=true , address=145.79.227.51:4410 , piece_id=YQF3IOBA7ZRBHXKZUHAYJEK5XKDTZGDT53LE4TLB52JRYTPPHKSA, seg_pos={0 0}, upload=303742]: Long tail cancellation due to optimal threshold May 13 23:53:30.967876334 (317h5m41.488700079s) [noise=true , address=77.90.31.15:4701 , piece_id=JJ65CNC4T5KDV5MRYDXMUCK5NFVJMREOTM7BR2LAJ3MW7YMDVQ6A, seg_pos={0 0}, upload=303742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF May 13 23:53:30.967880522 (317h5m41.488704267s) [noise=true , address=77.90.31.15:4701 , piece_id=JJ65CNC4T5KDV5MRYDXMUCK5NFVJMREOTM7BR2LAJ3MW7YMDVQ6A, seg_pos={0 0}, upload=303742]: Long tail cancellation due to optimal threshold May 13 23:53:30.967910258 (317h5m41.488734003s) [seg_pos={0 0}, upload=303742]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 13 23:53:30.967988959 (317h5m41.488812704s) [upload=303742]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 13 23:53:30.967996433 (317h5m41.488820178s) [upload=303742]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 13 23:53:31.260954856 (317h5m41.781778601s) [upload=303742]: Done uploading segments... May 13 23:53:31.260972500 (317h5m41.781796235s) [upload=303742]: Upload finished. err: <nil> Duration: 30.002419085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [................................XX....................XXXXXX] duration: 5.001158376s trace: https://jaeger.dev.storj.io/trace/356ee9ba46d6c357 FAIL (2026-05-13 23:53:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001239902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X...........XX...X..X.XX.....X.............X..X.....X..XXX.] duration: 21.838138524s trace: https://jaeger.dev.storj.io/trace/765ce647d42c6d38 FAIL (2026-05-13 23:52:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.838153803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.................................XX....................XXXXX] duration: 5.000708104s trace: https://jaeger.dev.storj.io/trace/2cf7141b1bc26fea FAIL (2026-05-13 23:52:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000753661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXX.X] duration: 30.242510392s trace: https://jaeger.dev.storj.io/trace/fd814360b435dba FAIL (2026-05-13 23:52:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 13 23:52:05.679734198 (317h4m16.200557943s) [upload=303726]: Starting upload May 13 23:52:05.679784855 (317h4m16.20060859s) [upload=303726]: Done starting upload May 13 23:52:05.679903682 (317h4m16.200727427s) [upload=303726]: Uploading segments... May 13 23:52:05.680035774 (317h4m16.200859519s) [upload=303726]: Got next segment. Inline: false May 13 23:52:05.680077524 (317h4m16.200901269s) [upload=303726]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:52:05.680082163 (317h4m16.200905908s) [upload=303726]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:52:05.916575697 (317h4m16.437399442s) [seg_pos={0 0}, upload=303726]: Begin upload segment... May 13 23:52:05.916623298 (317h4m16.437447043s) [seg_pos={0 0}, upload=303726]: Done begin upload segment. May 13 23:52:05.916648827 (317h4m16.437472572s) [noise=true , address=45.91.116.6:4974 , piece_id=GSR3NADQPACSO2A62IYW43MMYF62LEWDFHUBGCXQIF4PBNJVF6SA, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.916739991 (317h4m16.437563736s) [noise=true , address=145.79.227.27:4691 , piece_id=64Y4LGLCBEC2GSG6ZTWPLZZ2NBZURUF2XY57MDKLHY3DDKL7EL3A, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.917240946 (317h4m16.438064691s) [noise=true , address=145.79.227.34:4910 , piece_id=QWJ5PAL3UYYJEGBLHNKAVYVH7UYDYDCBHILCBBEF6YTHVII3FHCQ, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.917562510 (317h4m16.438386255s) [noise=true , address=145.79.227.5:4224 , piece_id=RWZDK3CWPKK524UDIZLY63ELKS424AKDUMOYQG36SQFVVKWW6CDQ, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.917907048 (317h4m16.438730793s) [noise=true , address=145.79.227.8:4333 , piece_id=76QSGI7MN6XAZTRNAE2MQ5X65ZKZVMLYGU6ZVLL6DRE5CJYCFGTQ, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.918198363 (317h4m16.439022109s) [noise=true , address=145.79.227.14:4839 , piece_id=GQGL5SKAI5AD33V7NV7ITW2YV2L2D5OXCTYDHYLYCPOPV4GSJCGQ, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.918492415 (317h4m16.43931616s) [noise=true , address=145.79.227.12:4771 , piece_id=W2CGFEMET3CNTCQXTXIIJKSJLD7SAZOSCHX722HNEM3MKLWHWC6A, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.918791896 (317h4m16.439615641s) [noise=true , address=145.79.227.32:4834 , piece_id=2OVFVRJR6TIDADV2UQ5LDHGMUYHO5FMV4RZ2H5ECEGTGXIDJM7IQ, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.919851859 (317h4m16.440675604s) [noise=true , address=145.79.227.3:4154 , piece_id=33QJ45D5EKOFPF66ZIDQGHERWEQL6MBPMDXYUXDRGKPLVRJRMCRA, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.919956860 (317h4m16.440780605s) [noise=true , address=145.79.227.21:4476 , piece_id=R7ZR6A3D6I6MCEGFSIEPOYNH3KSXPHURP5THWCWIP24OFO6ELZFQ, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.920938592 (317h4m16.441762337s) [noise=true , address=145.79.227.24:4585 , piece_id=HEPFKWSJ5BCMB3FAZFQULOBT22Y5REQOWNK3HH3YHNU2QYGSTKDA, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.922598761 (317h4m16.443422506s) [noise=true , address=145.79.227.6:4259 , piece_id=NTA2VTLMTJBU4HYKDOLRUNMX2E57CGDJCEUUOHM2UOWKA2C4OKNQ, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.923751891 (317h4m16.444575636s) [noise=true , address=145.79.227.46:4604 , piece_id=SJL2HMAQXP3VC7JN5HKVTVSIGCMTONFLRQBEJEAKVZRGT3477GHA, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.924959956 (317h4m16.445783701s) [noise=true , address=77.90.31.12:4594 , piece_id=7FER63T2K4SCLLAYASMMEARWNOSFXXHNBXVWYISO4COMQEONU3PA, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.925534983 (317h4m16.446358718s) [noise=true , address=145.79.227.31:4799 , piece_id=HHNXKCTBPPQPFHJLB6GQM3DWHIUKZH6GQRIV2WANFEACXBIGUECA, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:05.926502450 (317h4m16.447326195s) [noise=true , address=145.79.227.10:4402 , piece_id=DWRCW6VTRHNK5H2PSQOLSFUSCVB5DZMYNWMH77BMNQNAWXRVJS6Q, seg_pos={0 0}, upload=303726]: Uploading piece... May 13 23:52:08.172276711 (317h4m18.693100456s) [upload=303726]: Next returned nil segment May 13 23:52:08.172290707 (317h4m18.693114452s) [upload=303726]: Waiting for error group managing segments... May 13 23:52:08.548356560 (317h4m19.069180325s) [noise=true , address=45.91.116.6:4974 , piece_id=GSR3NADQPACSO2A62IYW43MMYF62LEWDFHUBGCXQIF4PBNJVF6SA, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:11.700902722 (317h4m22.221726467s) [noise=true , address=145.79.227.14:4839 , piece_id=GQGL5SKAI5AD33V7NV7ITW2YV2L2D5OXCTYDHYLYCPOPV4GSJCGQ, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:12.434869362 (317h4m22.955693107s) [noise=true , address=145.79.227.21:4476 , piece_id=R7ZR6A3D6I6MCEGFSIEPOYNH3KSXPHURP5THWCWIP24OFO6ELZFQ, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:12.498986162 (317h4m23.019809897s) [noise=true , address=145.79.227.24:4585 , piece_id=HEPFKWSJ5BCMB3FAZFQULOBT22Y5REQOWNK3HH3YHNU2QYGSTKDA, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:13.191923138 (317h4m23.712746883s) [noise=true , address=145.79.227.31:4799 , piece_id=HHNXKCTBPPQPFHJLB6GQM3DWHIUKZH6GQRIV2WANFEACXBIGUECA, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:13.209349362 (317h4m23.730173107s) [noise=true , address=145.79.227.10:4402 , piece_id=DWRCW6VTRHNK5H2PSQOLSFUSCVB5DZMYNWMH77BMNQNAWXRVJS6Q, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:14.557859165 (317h4m25.07868291s) [noise=true , address=145.79.227.12:4771 , piece_id=W2CGFEMET3CNTCQXTXIIJKSJLD7SAZOSCHX722HNEM3MKLWHWC6A, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:16.369575815 (317h4m26.89039956s) [noise=true , address=145.79.227.8:4333 , piece_id=76QSGI7MN6XAZTRNAE2MQ5X65ZKZVMLYGU6ZVLL6DRE5CJYCFGTQ, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:16.783491111 (317h4m27.304314856s) [noise=true , address=145.79.227.3:4154 , piece_id=33QJ45D5EKOFPF66ZIDQGHERWEQL6MBPMDXYUXDRGKPLVRJRMCRA, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:30.640810304 (317h4m41.161634049s) [noise=true , address=145.79.227.46:4604 , piece_id=SJL2HMAQXP3VC7JN5HKVTVSIGCMTONFLRQBEJEAKVZRGT3477GHA, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:31.559114061 (317h4m42.079937807s) [noise=true , address=145.79.227.5:4224 , piece_id=RWZDK3CWPKK524UDIZLY63ELKS424AKDUMOYQG36SQFVVKWW6CDQ, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:33.401330007 (317h4m43.922153742s) [noise=true , address=77.90.31.12:4594 , piece_id=7FER63T2K4SCLLAYASMMEARWNOSFXXHNBXVWYISO4COMQEONU3PA, seg_pos={0 0}, upload=303726]: Done uploading piece. err: <nil> May 13 23:52:35.016501391 (317h4m45.537325176s) [noise=true , address=145.79.227.34:4910 , piece_id=QWJ5PAL3UYYJEGBLHNKAVYVH7UYDYDCBHILCBBEF6YTHVII3FHCQ, seg_pos={0 0}, upload=303726]: Done uploading piece. err: ecclient: upload failed (node:12bRYzNggkq7QWFTP9fB94dTrMMX8AYvKMt2zjmJBzT1VwqyPVZ, address:145.79.227.34:4910): protocol: expected piece hash; context deadline exceeded; EOF May 13 23:52:35.016634474 (317h4m45.537458229s) [noise=true , address=145.79.227.32:4834 , piece_id=2OVFVRJR6TIDADV2UQ5LDHGMUYHO5FMV4RZ2H5ECEGTGXIDJM7IQ, seg_pos={0 0}, upload=303726]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:46378->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:46378->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46378->145.79.227.32:4834: use of closed network connection May 13 23:52:35.016677006 (317h4m45.537500751s) [noise=true , address=145.79.227.6:4259 , piece_id=NTA2VTLMTJBU4HYKDOLRUNMX2E57CGDJCEUUOHM2UOWKA2C4OKNQ, seg_pos={0 0}, upload=303726]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:42758->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:42758->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42758->145.79.227.6:4259: use of closed network connection May 13 23:52:35.016820910 (317h4m45.537644655s) [noise=true , address=145.79.227.27:4691 , piece_id=64Y4LGLCBEC2GSG6ZTWPLZZ2NBZURUF2XY57MDKLHY3DDKL7EL3A, seg_pos={0 0}, upload=303726]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF May 13 23:52:35.016852461 (317h4m45.537676216s) [seg_pos={0 0}, upload=303726]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 13 23:52:35.256520092 (317h4m45.777343847s) [upload=303726]: Done uploading segments... May 13 23:52:35.256560849 (317h4m45.777384594s) [upload=303726]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.242531292s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..X...........XX...X..X.XX.....X.............X..X.....X..XXX] duration: 1m0.000365168s trace: https://jaeger.dev.storj.io/trace/765ce647d42c6d38 FAIL (2026-05-13 23:52:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000376219s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..................................XX....................XXXX] duration: 5.000918886s trace: https://jaeger.dev.storj.io/trace/500f3cd22c6ef64d FAIL (2026-05-13 23:51:22 UTC) orbiter warning: context deadline exceeded Duration: 5.000931971s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 29.414728669s trace: https://jaeger.dev.storj.io/trace/3156d01900f804f5 FAIL (2026-05-13 23:51:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 13 23:50:20.651845795 (317h2m31.17266954s) [upload=303697]: Starting upload May 13 23:50:20.651979279 (317h2m31.172803044s) [upload=303697]: Done starting upload May 13 23:50:20.652094148 (317h2m31.172917924s) [upload=303697]: Uploading segments... May 13 23:50:20.652348072 (317h2m31.173171818s) [upload=303697]: Got next segment. Inline: false May 13 23:50:20.652393420 (317h2m31.173217175s) [upload=303697]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:50:20.652399852 (317h2m31.173223617s) [upload=303697]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:50:20.889114028 (317h2m31.409937773s) [seg_pos={0 0}, upload=303697]: Begin upload segment... May 13 23:50:20.889164984 (317h2m31.409988729s) [seg_pos={0 0}, upload=303697]: Done begin upload segment. May 13 23:50:20.889189732 (317h2m31.410013477s) [noise=true , address=145.79.227.22:4510 , piece_id=PM56MCCNDPWQAT47ISEA4WSYRRBBU6QFTGM57BRHKDBYTGTGJ2SQ, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.889201375 (317h2m31.41002511s) [noise=true , address=145.79.227.27:4695 , piece_id=A4FJHJPQP3F3F66KGLZ4YEA5I35N4IYSZ55SN6ZTQONM3GJSILRA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.889863849 (317h2m31.410687594s) [noise=true , address=145.79.227.21:4479 , piece_id=6FPH7UVZ5UHPR47IPO7XGQ7TJZT3MPMLYQ6FACAOMCJK7BIUZO3A, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.890025146 (317h2m31.410848891s) [noise=true , address=145.79.227.4:4188 , piece_id=Z2Q5LGEH45L2Q5RB7RBIWU7DSY4QUHOX4OGVIJ6S3I6J3DHY4FHQ, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.890594422 (317h2m31.411418167s) [noise=true , address=145.79.227.7:4299 , piece_id=QF4GCYW3U2CS463NBCTY6XQEGHIVVLDNTZF7VR3P4W7CLFELVUCQ, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.890599762 (317h2m31.411423508s) [noise=true , address=145.79.227.24:4583 , piece_id=3L7PIPIKVJHRBSD7NHAKGGB7KI5R5LGFXR7DVUPSMOS62IN5V7OA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.890849139 (317h2m31.411672884s) [noise=true , address=145.79.227.49:4338 , piece_id=SLXYRQSNWWKAWKEHHIPDQBVQNL6ABUOIHBPHNQJKQZADEN2E6QQA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.891075160 (317h2m31.411898905s) [noise=true , address=45.91.116.5:4297 , piece_id=X7PZMAQ5BKOHZUJ3LUBSPQMWNOKQGGS446S4AB3HHFC5HLI7YSQQ, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.891262878 (317h2m31.412086623s) [noise=true , address=45.91.116.7:4011 , piece_id=XPIXU4BEYBHSBBNXVKNIHKPU4QDNQ5NVXJDDNBFTHHCOFHFLH57Q, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.891632233 (317h2m31.412455978s) [noise=true , address=77.90.31.11:4553 , piece_id=K7KV7SHJTR2YU4HGQSLH4GCJQY753TSWE2QHUWPBVDLDAHHL7KXA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.892072713 (317h2m31.412896458s) [noise=true , address=145.79.227.18:4010 , piece_id=UYHWPNOYW63W3SCJTFGSCS4HWEUTAQOWM5KSWPRA2TKGR4LZGTNA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.892388887 (317h2m31.413212632s) [noise=true , address=145.79.227.20:4440 , piece_id=LP6ACKYNGFFMJA275RZMSSE67KJY43E35EHO4ACKFJNGG7TVGW7A, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.892619858 (317h2m31.413443603s) [noise=true , address=145.79.227.43:4158 , piece_id=DJMBPBUN3XV4LAXMP3HKQHIRPAHFXFQYAAVBNZALXHD64DV3TFYA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.892966299 (317h2m31.413790044s) [noise=true , address=145.79.227.32:4834 , piece_id=KNKO5XJ5GRJKBHV4HJFUJX3CYJE3SBOT3UIYEEQEI7KTWMPDT7HA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.893107518 (317h2m31.413931253s) [noise=true , address=45.91.116.4:4940 , piece_id=PCDV4HNAYVCL6VHB6M346HCH6E3MSNEZBP3VNU6PCHMC53NMU5CA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.893494567 (317h2m31.414318312s) [noise=true , address=145.79.227.8:4333 , piece_id=TEQL3MINJC7SDQUHCHQ5QULS345GFZVCAZ4RIDNPYWDEEMVFYB7A, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:23.130883959 (317h2m33.651707704s) [upload=303697]: Next returned nil segment May 13 23:50:23.130899569 (317h2m33.651723314s) [upload=303697]: Waiting for error group managing segments... May 13 23:50:23.494908883 (317h2m34.015732628s) [noise=true , address=45.91.116.4:4940 , piece_id=PCDV4HNAYVCL6VHB6M346HCH6E3MSNEZBP3VNU6PCHMC53NMU5CA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:23.584098147 (317h2m34.104921892s) [noise=true , address=45.91.116.7:4011 , piece_id=XPIXU4BEYBHSBBNXVKNIHKPU4QDNQ5NVXJDDNBFTHHCOFHFLH57Q, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:23.625277276 (317h2m34.146101021s) [noise=true , address=45.91.116.5:4297 , piece_id=X7PZMAQ5BKOHZUJ3LUBSPQMWNOKQGGS446S4AB3HHFC5HLI7YSQQ, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:28.099787323 (317h2m38.620611059s) [noise=true , address=145.79.227.24:4583 , piece_id=3L7PIPIKVJHRBSD7NHAKGGB7KI5R5LGFXR7DVUPSMOS62IN5V7OA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:32.511248690 (317h2m43.032072435s) [noise=true , address=145.79.227.18:4010 , piece_id=UYHWPNOYW63W3SCJTFGSCS4HWEUTAQOWM5KSWPRA2TKGR4LZGTNA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:33.460858276 (317h2m43.981682021s) [noise=true , address=145.79.227.43:4158 , piece_id=DJMBPBUN3XV4LAXMP3HKQHIRPAHFXFQYAAVBNZALXHD64DV3TFYA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:36.102618929 (317h2m46.623442674s) [noise=true , address=145.79.227.32:4834 , piece_id=KNKO5XJ5GRJKBHV4HJFUJX3CYJE3SBOT3UIYEEQEI7KTWMPDT7HA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:37.831934218 (317h2m48.352757963s) [noise=true , address=145.79.227.22:4510 , piece_id=PM56MCCNDPWQAT47ISEA4WSYRRBBU6QFTGM57BRHKDBYTGTGJ2SQ, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:40.266604778 (317h2m50.787428523s) [noise=true , address=145.79.227.20:4440 , piece_id=LP6ACKYNGFFMJA275RZMSSE67KJY43E35EHO4ACKFJNGG7TVGW7A, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:45.923534177 (317h2m56.444357922s) [noise=true , address=145.79.227.8:4333 , piece_id=TEQL3MINJC7SDQUHCHQ5QULS345GFZVCAZ4RIDNPYWDEEMVFYB7A, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:47.363033036 (317h2m57.883856781s) [noise=true , address=145.79.227.4:4188 , piece_id=Z2Q5LGEH45L2Q5RB7RBIWU7DSY4QUHOX4OGVIJ6S3I6J3DHY4FHQ, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:49.113769209 (317h2m59.634592964s) [noise=true , address=145.79.227.49:4338 , piece_id=SLXYRQSNWWKAWKEHHIPDQBVQNL6ABUOIHBPHNQJKQZADEN2E6QQA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:49.987676006 (317h3m0.508499752s) [noise=true , address=77.90.31.11:4553 , piece_id=K7KV7SHJTR2YU4HGQSLH4GCJQY753TSWE2QHUWPBVDLDAHHL7KXA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:46918->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:46918->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46918->77.90.31.11:4553: use of closed network connection May 13 23:50:49.987715613 (317h3m0.508539358s) [noise=true , address=145.79.227.21:4479 , piece_id=6FPH7UVZ5UHPR47IPO7XGQ7TJZT3MPMLYQ6FACAOMCJK7BIUZO3A, seg_pos={0 0}, upload=303697]: Done uploading piece. err: ecclient: upload failed (node:12VCopFEoHk99GsQNQPuWvTQZRD1bpXi8HHLKXaXRdvFThTxmAk, address:145.79.227.21:4479): protocol: expected piece hash; context deadline exceeded; EOF May 13 23:50:49.987778503 (317h3m0.508602258s) [noise=true , address=145.79.227.27:4695 , piece_id=A4FJHJPQP3F3F66KGLZ4YEA5I35N4IYSZ55SN6ZTQONM3GJSILRA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: ecclient: upload failed (node:1scGuKGuDJCYuUGp5nDdU1Sxr23L7EEPChWtLAhZ52i3RpiF6y, address:145.79.227.27:4695): protocol: write tcp 188.166.225.170:53712->145.79.227.27:4695: use of closed network connection; write tcp 188.166.225.170:53712->145.79.227.27:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53712->145.79.227.27:4695: use of closed network connection May 13 23:50:49.987811666 (317h3m0.508635411s) [noise=true , address=145.79.227.7:4299 , piece_id=QF4GCYW3U2CS463NBCTY6XQEGHIVVLDNTZF7VR3P4W7CLFELVUCQ, seg_pos={0 0}, upload=303697]: Done uploading piece. err: ecclient: upload failed (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n, address:145.79.227.7:4299): protocol: write tcp 188.166.225.170:53228->145.79.227.7:4299: use of closed network connection; write tcp 188.166.225.170:53228->145.79.227.7:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53228->145.79.227.7:4299: use of closed network connection May 13 23:50:49.987832245 (317h3m0.50865599s) [seg_pos={0 0}, upload=303697]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 13 23:50:50.225363490 (317h3m0.746187235s) [upload=303697]: Done uploading segments... May 13 23:50:50.225401883 (317h3m0.746225628s) [upload=303697]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 29.41474503s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...X...........XX...X..X.XX.....X.............X..X.....X..XX] duration: 1m0.000651142s trace: https://jaeger.dev.storj.io/trace/42c4b715e03f0a3e FAIL (2026-05-13 23:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000660199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................................XX....................XXX] duration: 5.001561345s trace: https://jaeger.dev.storj.io/trace/41432a0b2f71ef04 FAIL (2026-05-13 23:50:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001576404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.239085555s trace: https://jaeger.dev.storj.io/trace/3156d01900f804f5 FAIL (2026-05-13 23:50:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 13 23:50:20.651845795 (317h2m31.17266954s) [upload=303697]: Starting upload May 13 23:50:20.651979279 (317h2m31.172803044s) [upload=303697]: Done starting upload May 13 23:50:20.652094148 (317h2m31.172917924s) [upload=303697]: Uploading segments... May 13 23:50:20.652348072 (317h2m31.173171818s) [upload=303697]: Got next segment. Inline: false May 13 23:50:20.652393420 (317h2m31.173217175s) [upload=303697]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 13 23:50:20.652399852 (317h2m31.173223617s) [upload=303697]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 13 23:50:20.889114028 (317h2m31.409937773s) [seg_pos={0 0}, upload=303697]: Begin upload segment... May 13 23:50:20.889164984 (317h2m31.409988729s) [seg_pos={0 0}, upload=303697]: Done begin upload segment. May 13 23:50:20.889189732 (317h2m31.410013477s) [noise=true , address=145.79.227.22:4510 , piece_id=PM56MCCNDPWQAT47ISEA4WSYRRBBU6QFTGM57BRHKDBYTGTGJ2SQ, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.889201375 (317h2m31.41002511s) [noise=true , address=145.79.227.27:4695 , piece_id=A4FJHJPQP3F3F66KGLZ4YEA5I35N4IYSZ55SN6ZTQONM3GJSILRA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.889863849 (317h2m31.410687594s) [noise=true , address=145.79.227.21:4479 , piece_id=6FPH7UVZ5UHPR47IPO7XGQ7TJZT3MPMLYQ6FACAOMCJK7BIUZO3A, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.890025146 (317h2m31.410848891s) [noise=true , address=145.79.227.4:4188 , piece_id=Z2Q5LGEH45L2Q5RB7RBIWU7DSY4QUHOX4OGVIJ6S3I6J3DHY4FHQ, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.890594422 (317h2m31.411418167s) [noise=true , address=145.79.227.7:4299 , piece_id=QF4GCYW3U2CS463NBCTY6XQEGHIVVLDNTZF7VR3P4W7CLFELVUCQ, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.890599762 (317h2m31.411423508s) [noise=true , address=145.79.227.24:4583 , piece_id=3L7PIPIKVJHRBSD7NHAKGGB7KI5R5LGFXR7DVUPSMOS62IN5V7OA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.890849139 (317h2m31.411672884s) [noise=true , address=145.79.227.49:4338 , piece_id=SLXYRQSNWWKAWKEHHIPDQBVQNL6ABUOIHBPHNQJKQZADEN2E6QQA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.891075160 (317h2m31.411898905s) [noise=true , address=45.91.116.5:4297 , piece_id=X7PZMAQ5BKOHZUJ3LUBSPQMWNOKQGGS446S4AB3HHFC5HLI7YSQQ, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.891262878 (317h2m31.412086623s) [noise=true , address=45.91.116.7:4011 , piece_id=XPIXU4BEYBHSBBNXVKNIHKPU4QDNQ5NVXJDDNBFTHHCOFHFLH57Q, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.891632233 (317h2m31.412455978s) [noise=true , address=77.90.31.11:4553 , piece_id=K7KV7SHJTR2YU4HGQSLH4GCJQY753TSWE2QHUWPBVDLDAHHL7KXA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.892072713 (317h2m31.412896458s) [noise=true , address=145.79.227.18:4010 , piece_id=UYHWPNOYW63W3SCJTFGSCS4HWEUTAQOWM5KSWPRA2TKGR4LZGTNA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.892388887 (317h2m31.413212632s) [noise=true , address=145.79.227.20:4440 , piece_id=LP6ACKYNGFFMJA275RZMSSE67KJY43E35EHO4ACKFJNGG7TVGW7A, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.892619858 (317h2m31.413443603s) [noise=true , address=145.79.227.43:4158 , piece_id=DJMBPBUN3XV4LAXMP3HKQHIRPAHFXFQYAAVBNZALXHD64DV3TFYA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.892966299 (317h2m31.413790044s) [noise=true , address=145.79.227.32:4834 , piece_id=KNKO5XJ5GRJKBHV4HJFUJX3CYJE3SBOT3UIYEEQEI7KTWMPDT7HA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.893107518 (317h2m31.413931253s) [noise=true , address=45.91.116.4:4940 , piece_id=PCDV4HNAYVCL6VHB6M346HCH6E3MSNEZBP3VNU6PCHMC53NMU5CA, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:20.893494567 (317h2m31.414318312s) [noise=true , address=145.79.227.8:4333 , piece_id=TEQL3MINJC7SDQUHCHQ5QULS345GFZVCAZ4RIDNPYWDEEMVFYB7A, seg_pos={0 0}, upload=303697]: Uploading piece... May 13 23:50:23.130883959 (317h2m33.651707704s) [upload=303697]: Next returned nil segment May 13 23:50:23.130899569 (317h2m33.651723314s) [upload=303697]: Waiting for error group managing segments... May 13 23:50:23.494908883 (317h2m34.015732628s) [noise=true , address=45.91.116.4:4940 , piece_id=PCDV4HNAYVCL6VHB6M346HCH6E3MSNEZBP3VNU6PCHMC53NMU5CA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:23.584098147 (317h2m34.104921892s) [noise=true , address=45.91.116.7:4011 , piece_id=XPIXU4BEYBHSBBNXVKNIHKPU4QDNQ5NVXJDDNBFTHHCOFHFLH57Q, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:23.625277276 (317h2m34.146101021s) [noise=true , address=45.91.116.5:4297 , piece_id=X7PZMAQ5BKOHZUJ3LUBSPQMWNOKQGGS446S4AB3HHFC5HLI7YSQQ, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:28.099787323 (317h2m38.620611059s) [noise=true , address=145.79.227.24:4583 , piece_id=3L7PIPIKVJHRBSD7NHAKGGB7KI5R5LGFXR7DVUPSMOS62IN5V7OA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:32.511248690 (317h2m43.032072435s) [noise=true , address=145.79.227.18:4010 , piece_id=UYHWPNOYW63W3SCJTFGSCS4HWEUTAQOWM5KSWPRA2TKGR4LZGTNA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:33.460858276 (317h2m43.981682021s) [noise=true , address=145.79.227.43:4158 , piece_id=DJMBPBUN3XV4LAXMP3HKQHIRPAHFXFQYAAVBNZALXHD64DV3TFYA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:36.102618929 (317h2m46.623442674s) [noise=true , address=145.79.227.32:4834 , piece_id=KNKO5XJ5GRJKBHV4HJFUJX3CYJE3SBOT3UIYEEQEI7KTWMPDT7HA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:37.831934218 (317h2m48.352757963s) [noise=true , address=145.79.227.22:4510 , piece_id=PM56MCCNDPWQAT47ISEA4WSYRRBBU6QFTGM57BRHKDBYTGTGJ2SQ, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:40.266604778 (317h2m50.787428523s) [noise=true , address=145.79.227.20:4440 , piece_id=LP6ACKYNGFFMJA275RZMSSE67KJY43E35EHO4ACKFJNGG7TVGW7A, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:45.923534177 (317h2m56.444357922s) [noise=true , address=145.79.227.8:4333 , piece_id=TEQL3MINJC7SDQUHCHQ5QULS345GFZVCAZ4RIDNPYWDEEMVFYB7A, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:47.363033036 (317h2m57.883856781s) [noise=true , address=145.79.227.4:4188 , piece_id=Z2Q5LGEH45L2Q5RB7RBIWU7DSY4QUHOX4OGVIJ6S3I6J3DHY4FHQ, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:49.113769209 (317h2m59.634592964s) [noise=true , address=145.79.227.49:4338 , piece_id=SLXYRQSNWWKAWKEHHIPDQBVQNL6ABUOIHBPHNQJKQZADEN2E6QQA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: <nil> May 13 23:50:49.987676006 (317h3m0.508499752s) [noise=true , address=77.90.31.11:4553 , piece_id=K7KV7SHJTR2YU4HGQSLH4GCJQY753TSWE2QHUWPBVDLDAHHL7KXA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:46918->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:46918->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46918->77.90.31.11:4553: use of closed network connection May 13 23:50:49.987715613 (317h3m0.508539358s) [noise=true , address=145.79.227.21:4479 , piece_id=6FPH7UVZ5UHPR47IPO7XGQ7TJZT3MPMLYQ6FACAOMCJK7BIUZO3A, seg_pos={0 0}, upload=303697]: Done uploading piece. err: ecclient: upload failed (node:12VCopFEoHk99GsQNQPuWvTQZRD1bpXi8HHLKXaXRdvFThTxmAk, address:145.79.227.21:4479): protocol: expected piece hash; context deadline exceeded; EOF May 13 23:50:49.987778503 (317h3m0.508602258s) [noise=true , address=145.79.227.27:4695 , piece_id=A4FJHJPQP3F3F66KGLZ4YEA5I35N4IYSZ55SN6ZTQONM3GJSILRA, seg_pos={0 0}, upload=303697]: Done uploading piece. err: ecclient: upload failed (node:1scGuKGuDJCYuUGp5nDdU1Sxr23L7EEPChWtLAhZ52i3RpiF6y, address:145.79.227.27:4695): protocol: write tcp 188.166.225.170:53712->145.79.227.27:4695: use of closed network connection; write tcp 188.166.225.170:53712->145.79.227.27:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53712->145.79.227.27:4695: use of closed network connection May 13 23:50:49.987811666 (317h3m0.508635411s) [noise=true , address=145.79.227.7:4299 , piece_id=QF4GCYW3U2CS463NBCTY6XQEGHIVVLDNTZF7VR3P4W7CLFELVUCQ, seg_pos={0 0}, upload=303697]: Done uploading piece. err: ecclient: upload failed (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n, address:145.79.227.7:4299): protocol: write tcp 188.166.225.170:53228->145.79.227.7:4299: use of closed network connection; write tcp 188.166.225.170:53228->145.79.227.7:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53228->145.79.227.7:4299: use of closed network connection May 13 23:50:49.987832245 (317h3m0.50865599s) [seg_pos={0 0}, upload=303697]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 13 23:50:50.225363490 (317h3m0.746187235s) [upload=303697]: Done uploading segments... May 13 23:50:50.225401883 (317h3m0.746225628s) [upload=303697]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.239122665s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [....X...........XX...X..X.XX.....X.............X..X.....X..X] duration: 1m0.000405405s trace: https://jaeger.dev.storj.io/trace/6201c92e65085530 FAIL (2026-05-13 23:50:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000424191s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................................XX....................XX] duration: 5.000763283s trace: https://jaeger.dev.storj.io/trace/12411821e7e2a67c FAIL (2026-05-13 23:50:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000785105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.238256414s trace: https://jaeger.dev.storj.io/trace/7cc5ebf469e0ae0d FAIL (2026-05-13 23:50:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :40.936702530 (317h1m51.457526275s) [noise=true , address=45.91.116.7:4010 , piece_id=3WGWYCEXKQRY2CVTTLKB3DBWL3FHHD3ZYZIHWBWE6MHWLVB4U2PQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.937035455 (317h1m51.4578592s) [noise=true , address=77.90.31.17:4771 , piece_id=KWYLQVOTHTMOPPNQFR7FHFVQIHKOJBG4V6O3DYRUKMB7R3XYRX5Q, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.937377027 (317h1m51.458200772s) [noise=true , address=145.79.227.39:4018 , piece_id=VSWDFJL4MBTNUVTO2XFJHVYZQBYRVF7BVYHNQ7BAJH7K6K4JLBKA, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.937717236 (317h1m51.458540981s) [noise=true , address=145.79.227.27:4695 , piece_id=AXWNKS6LBE6E6QJLQ5GJGBNY64XYHT4LFDFSR3OKPZHEFCP4XKFQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.938038409 (317h1m51.458862155s) [noise=true , address=145.79.227.11:4729 , piece_id=DIKUV55MA347GILRM4BT4SBJ4KAYQQE74JEEEFPNQFWELI2SCPCQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.939173345 (317h1m51.45999709s) [noise=true , address=145.79.227.47:4271 , piece_id=FUAME7AU4IM73EKNSEKGTZQCMEI2SIK5LNQRRUFTDOOBEY7RV2WQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.939185508 (317h1m51.460009253s) [noise=true , address=145.79.227.28:4729 , piece_id=RMWZRWDU4QYZM2XORGEHDNCKBVWJ3YKHFCD277LZ2RUUG5U6CXWQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.940361462 (317h1m51.461185207s) [noise=true , address=145.79.227.24:4585 , piece_id=T3SLZUGLECT56CCU2T4AMHCONHKBB37RPEGUKHJZWRWRLICBZO6A, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.942109598 (317h1m51.462933343s) [noise=true , address=45.91.116.10:4088 , piece_id=NMCQDNGOLQWOO247FY5HZEF6HI24SDUMBCAJ7LGDNRETLYD6WOVQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.943527343 (317h1m51.464351088s) [noise=true , address=145.79.227.7:4299 , piece_id=ITNQ5TWGEV6PTGTQSKRDSI3NVWURG7IKVRFHAQ2RZJE2IM3O6JZQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.944766799 (317h1m51.465590544s) [noise=true , address=145.79.227.14:4837 , piece_id=V6XEB6VEXPIXJZHKJL5SLRISTRJIWXPNYYDRPWUUBBB7IUROVPQQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.946148105 (317h1m51.46697185s) [noise=true , address=45.91.116.12:4226 , piece_id=NW4DEURJQ5M4CRQWKKTTK35T6AW42NNNXY2TZUO4XF3QXITT4DAQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:40.948749208 (317h1m51.469572953s) [noise=true , address=145.79.227.18:4010 , piece_id=INUFKGXYX6CMV6JKJPZK6J42NTQOI3KXEXQYUYKTZ3X3HA7PDTSQ, seg_pos={0 0}, upload=303686]: Uploading piece... May 13 23:49:43.223745313 (317h1m53.744569048s) [upload=303686]: Next returned nil segment May 13 23:49:43.223767326 (317h1m53.744591071s) [upload=303686]: Waiting for error group managing segments... May 13 23:49:43.571592627 (317h1m54.092416362s) [noise=true , address=45.91.116.10:4088 , piece_id=NMCQDNGOLQWOO247FY5HZEF6HI24SDUMBCAJ7LGDNRETLYD6WOVQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: <nil> May 13 23:49:43.610772743 (317h1m54.131596488s) [noise=true , address=45.91.116.12:4226 , piece_id=NW4DEURJQ5M4CRQWKKTTK35T6AW42NNNXY2TZUO4XF3QXITT4DAQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: <nil> May 13 23:49:43.675051571 (317h1m54.195875316s) [noise=true , address=45.91.116.7:4010 , piece_id=3WGWYCEXKQRY2CVTTLKB3DBWL3FHHD3ZYZIHWBWE6MHWLVB4U2PQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: <nil> May 13 23:49:48.466196023 (317h1m58.987019768s) [noise=true , address=145.79.227.27:4695 , piece_id=AXWNKS6LBE6E6QJLQ5GJGBNY64XYHT4LFDFSR3OKPZHEFCP4XKFQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: <nil> May 13 23:49:56.746982259 (317h2m7.267806004s) [noise=true , address=145.79.227.46:4602 , piece_id=QC4D6WMLCDGMHKVM524ZEBLTV6X5LEEV6UPYAU2UQUJLWCZ5DZJA, seg_pos={0 0}, upload=303686]: Done uploading piece. err: <nil> May 13 23:49:59.862717642 (317h2m10.383541387s) [noise=true , address=77.90.31.19:4846 , piece_id=ZQOFIY73Y24B3TBE7UGCZRP25PSLCPPFHZOQYTUUPA3T4NR4D33A, seg_pos={0 0}, upload=303686]: Done uploading piece. err: <nil> May 13 23:50:05.581558703 (317h2m16.102382448s) [noise=true , address=145.79.227.14:4837 , piece_id=V6XEB6VEXPIXJZHKJL5SLRISTRJIWXPNYYDRPWUUBBB7IUROVPQQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: <nil> May 13 23:50:09.938915317 (317h2m20.459739062s) [noise=true , address=145.79.227.18:4010 , piece_id=INUFKGXYX6CMV6JKJPZK6J42NTQOI3KXEXQYUYKTZ3X3HA7PDTSQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: <nil> May 13 23:50:10.025786327 (317h2m20.546610083s) [noise=true , address=145.79.227.7:4299 , piece_id=ITNQ5TWGEV6PTGTQSKRDSI3NVWURG7IKVRFHAQ2RZJE2IM3O6JZQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: ecclient: upload failed (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n, address:145.79.227.7:4299): protocol: write tcp 188.166.225.170:34432->145.79.227.7:4299: use of closed network connection; write tcp 188.166.225.170:34432->145.79.227.7:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34432->145.79.227.7:4299: use of closed network connection May 13 23:50:10.025830251 (317h2m20.546653996s) [noise=true , address=77.90.31.17:4771 , piece_id=KWYLQVOTHTMOPPNQFR7FHFVQIHKOJBG4V6O3DYRUKMB7R3XYRX5Q, seg_pos={0 0}, upload=303686]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:38976->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:38976->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38976->77.90.31.17:4771: use of closed network connection May 13 23:50:10.025894493 (317h2m20.546718238s) [noise=true , address=145.79.227.39:4018 , piece_id=VSWDFJL4MBTNUVTO2XFJHVYZQBYRVF7BVYHNQ7BAJH7K6K4JLBKA, seg_pos={0 0}, upload=303686]: Done uploading piece. err: ecclient: upload failed (node:12YcYgDWq2k7F9fwpdLAGNkrskGEUURe6QuRed7mRjMTdNHEQ8D, address:145.79.227.39:4018): protocol: write tcp 188.166.225.170:52974->145.79.227.39:4018: use of closed network connection; write tcp 188.166.225.170:52974->145.79.227.39:4018: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52974->145.79.227.39:4018: use of closed network connection May 13 23:50:10.025924350 (317h2m20.546748095s) [noise=true , address=145.79.227.47:4271 , piece_id=FUAME7AU4IM73EKNSEKGTZQCMEI2SIK5LNQRRUFTDOOBEY7RV2WQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: ecclient: upload failed (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1, address:145.79.227.47:4271): protocol: write tcp 188.166.225.170:49614->145.79.227.47:4271: use of closed network connection; write tcp 188.166.225.170:49614->145.79.227.47:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49614->145.79.227.47:4271: use of closed network connection May 13 23:50:10.025962232 (317h2m20.546785977s) [noise=true , address=145.79.227.11:4729 , piece_id=DIKUV55MA347GILRM4BT4SBJ4KAYQQE74JEEEFPNQFWELI2SCPCQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:55572->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:55572->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55572->145.79.227.11:4729: use of closed network connection May 13 23:50:10.026025734 (317h2m20.546849479s) [noise=true , address=145.79.227.28:4729 , piece_id=RMWZRWDU4QYZM2XORGEHDNCKBVWJ3YKHFCD277LZ2RUUG5U6CXWQ, seg_pos={0 0}, upload=303686]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: write tcp 188.166.225.170:37664->145.79.227.28:4729: use of closed network connection; write tcp 188.166.225.170:37664->145.79.227.28:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37664->145.79.227.28:4729: use of closed network connection May 13 23:50:10.026165480 (317h2m20.546989225s) [noise=true , address=145.79.227.24:4585 , piece_id=T3SLZUGLECT56CCU2T4AMHCONHKBB37RPEGUKHJZWRWRLICBZO6A, seg_pos={0 0}, upload=303686]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: write tcp 188.166.225.170:58148->145.79.227.24:4585: use of closed network connection; write tcp 188.166.225.170:58148->145.79.227.24:4585: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58148->145.79.227.24:4585: use of closed network connection May 13 23:50:10.026266634 (317h2m20.547090379s) [noise=true , address=145.79.227.22:4511 , piece_id=2BH6C4X5VR7B4YVEZEIVHS2BAK6F2PT7RDRTPIOAZN6QYQ6P4OOA, seg_pos={0 0}, upload=303686]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:45426->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:45426->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45426->145.79.227.22:4511: use of closed network connection May 13 23:50:10.026290839 (317h2m20.547114584s) [seg_pos={0 0}, upload=303686]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> May 13 23:50:10.263228111 (317h2m20.784051856s) [upload=303686]: Done uploading segments... May 13 23:50:10.263257557 (317h2m20.784081302s) [upload=303686]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.238265661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.265283211s trace: https://jaeger.dev.storj.io/trace/605602d4c4c718e6 FAIL (2026-05-13 23:49:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ad=303679]: Uploading piece... May 13 23:49:10.451656042 (317h1m20.972479787s) [noise=true , address=145.79.227.13:4802 , piece_id=CENFQEG5XFHFWBJPKGOJPEJNAL7R2OLBU5GNSTQRX3GYI4II2LEA, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.451715335 (317h1m20.97253908s) [noise=true , address=77.90.31.13:4627 , piece_id=EESPH4ICVGKMFQDJDQGFLLZLFXY4Y4C2WAIHNHMZGY36XG6ZDKHQ, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.451896010 (317h1m20.972719755s) [noise=true , address=145.79.227.51:4412 , piece_id=TVH2RWAJR6XHHDHZF27APICHUKPHLULA7237ZBLE4OVF46VOVOEA, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.451992363 (317h1m20.972816108s) [noise=true , address=145.79.227.32:4838 , piece_id=Z7KUZCZMDXM7DBZAUZJUU2YWOTAKRQFGPINHNHA34IGP5LFOXXSQ, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.452158020 (317h1m20.972981755s) [noise=true , address=45.91.116.11:4117 , piece_id=NXKNR5HVNWRMBPRW4SGUDIQXBIDB7GHU2SVQAMIACVALOFOGHMSA, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.452246077 (317h1m20.973069822s) [noise=true , address=145.79.227.14:4840 , piece_id=2U7UYXD7Z3UCTAX42CBGHJEHQI7F32F32E7IIMR2YSHBFP52Y4PQ, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.452375184 (317h1m20.973198929s) [noise=true , address=145.79.227.30:4761 , piece_id=2MQ6E46F4THQ4EQKH2BX5F3FZJJ57657YKXRWJZWGS7FQPX4G2CA, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.452472790 (317h1m20.973296525s) [noise=true , address=45.91.116.8:4079 , piece_id=GHYPC3OCUCA6IGTZJUEMKDHJ7OMQKIZC5MQA62Q4IGYN5OOKOWIA, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.452762884 (317h1m20.973586619s) [noise=true , address=145.79.227.43:4161 , piece_id=CMTC6YTMOL2EJJSVZYUPLCYPRIWAZ7NXPI7QPDB5QTTB2FWRP3GQ, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:10.453116127 (317h1m20.973939862s) [noise=true , address=145.79.227.48:4304 , piece_id=HMHGICHYZTTPIL5YK724IDZ2EQ3ZYQ2ZNWR3ZGEGIYYTUGJGUHPQ, seg_pos={0 0}, upload=303679]: Uploading piece... May 13 23:49:12.678299524 (317h1m23.199123269s) [upload=303679]: Next returned nil segment May 13 23:49:12.678312910 (317h1m23.199136646s) [upload=303679]: Waiting for error group managing segments... May 13 23:49:13.030404934 (317h1m23.551228679s) [noise=true , address=45.91.116.5:4300 , piece_id=ZG5J5FFE46LH5STZQNK7OM6AAYT4PFVE72SFNHWOQKM4WDFBPJ3A, seg_pos={0 0}, upload=303679]: Done uploading piece. err: <nil> May 13 23:49:13.109293591 (317h1m23.630117336s) [noise=true , address=45.91.116.11:4117 , piece_id=NXKNR5HVNWRMBPRW4SGUDIQXBIDB7GHU2SVQAMIACVALOFOGHMSA, seg_pos={0 0}, upload=303679]: Done uploading piece. err: <nil> May 13 23:49:14.028971172 (317h1m24.549794917s) [noise=true , address=45.91.116.14:4190 , piece_id=AY5CQFAJX23M4TAOTEVJ4VFQLGHJG2BEC4SS2G3WAZSVOU4U42QQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: <nil> May 13 23:49:14.088857780 (317h1m24.609681525s) [noise=true , address=45.91.116.8:4079 , piece_id=GHYPC3OCUCA6IGTZJUEMKDHJ7OMQKIZC5MQA62Q4IGYN5OOKOWIA, seg_pos={0 0}, upload=303679]: Done uploading piece. err: <nil> May 13 23:49:22.593765974 (317h1m33.114589709s) [noise=true , address=145.79.227.4:4185 , piece_id=6AHLGTGMUBCKV7TVYVCPSXBLI3MGK7YZSU4HWHVMFU5LZ3X2UMFA, seg_pos={0 0}, upload=303679]: Done uploading piece. err: <nil> May 13 23:49:27.834774681 (317h1m38.355598416s) [noise=true , address=145.79.227.14:4840 , piece_id=2U7UYXD7Z3UCTAX42CBGHJEHQI7F32F32E7IIMR2YSHBFP52Y4PQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: <nil> May 13 23:49:39.534281553 (317h1m50.055105299s) [noise=true , address=145.79.227.13:4802 , piece_id=CENFQEG5XFHFWBJPKGOJPEJNAL7R2OLBU5GNSTQRX3GYI4II2LEA, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:40040->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40040->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->145.79.227.13:4802: use of closed network connection May 13 23:49:39.534329856 (317h1m50.055153601s) [noise=true , address=145.79.227.51:4412 , piece_id=TVH2RWAJR6XHHDHZF27APICHUKPHLULA7237ZBLE4OVF46VOVOEA, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:12vBvEfCYZxsv17cVsQTbCQ4YjZfiD4RXjWyz7Ka9ZZrtvzC7SS, address:145.79.227.51:4412): protocol: expected piece hash; context deadline exceeded; EOF May 13 23:49:39.534374030 (317h1m50.055197765s) [noise=true , address=145.79.227.22:4514 , piece_id=AZSMZMDVHT53VJWBELGLE52DV2URUPL47JOUMJFZSJF5AHGZIFPQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:145.79.227.22:4514): protocol: write tcp 188.166.225.170:58448->145.79.227.22:4514: use of closed network connection; write tcp 188.166.225.170:58448->145.79.227.22:4514: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58448->145.79.227.22:4514: use of closed network connection May 13 23:49:39.534417242 (317h1m50.055240987s) [noise=true , address=145.79.227.32:4838 , piece_id=Z7KUZCZMDXM7DBZAUZJUU2YWOTAKRQFGPINHNHA34IGP5LFOXXSQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6, address:145.79.227.32:4838): protocol: write tcp 188.166.225.170:55554->145.79.227.32:4838: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.32:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.32:4838: use of closed network connection May 13 23:49:39.534463150 (317h1m50.055286895s) [noise=true , address=145.79.227.45:4233 , piece_id=RMOHIIRT7UVLKA5OFUM4CHSIXR6GRHRIOMEQ5WPMSKUHDUPYVWNQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:145.79.227.45:4233): protocol: write tcp 188.166.225.170:48208->145.79.227.45:4233: use of closed network connection; write tcp 188.166.225.170:48208->145.79.227.45:4233: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48208->145.79.227.45:4233: use of closed network connection May 13 23:49:39.534532772 (317h1m50.055356518s) [noise=true , address=145.79.227.8:4331 , piece_id=FZORRLF7NS4DISYHRIWRSSWK6ZNKCSID7NNUHYAWIC2KLRDSCADQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:45922->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:45922->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45922->145.79.227.8:4331: use of closed network connection May 13 23:49:39.534609910 (317h1m50.055433655s) [noise=true , address=77.90.31.13:4627 , piece_id=EESPH4ICVGKMFQDJDQGFLLZLFXY4Y4C2WAIHNHMZGY36XG6ZDKHQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:34528->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:34528->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34528->77.90.31.13:4627: use of closed network connection May 13 23:49:39.534672650 (317h1m50.055496395s) [noise=true , address=145.79.227.30:4761 , piece_id=2MQ6E46F4THQ4EQKH2BX5F3FZJJ57657YKXRWJZWGS7FQPX4G2CA, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:35424->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:35424->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35424->145.79.227.30:4761: use of closed network connection May 13 23:49:39.534741581 (317h1m50.055565316s) [noise=true , address=145.79.227.43:4161 , piece_id=CMTC6YTMOL2EJJSVZYUPLCYPRIWAZ7NXPI7QPDB5QTTB2FWRP3GQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD, address:145.79.227.43:4161): protocol: write tcp 188.166.225.170:60076->145.79.227.43:4161: use of closed network connection; write tcp 188.166.225.170:60076->145.79.227.43:4161: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60076->145.79.227.43:4161: use of closed network connection May 13 23:49:39.534802537 (317h1m50.055626272s) [noise=true , address=145.79.227.48:4304 , piece_id=HMHGICHYZTTPIL5YK724IDZ2EQ3ZYQ2ZNWR3ZGEGIYYTUGJGUHPQ, seg_pos={0 0}, upload=303679]: Done uploading piece. err: ecclient: upload failed (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG, address:145.79.227.48:4304): protocol: write tcp 188.166.225.170:54452->145.79.227.48:4304: use of closed network connection; write tcp 188.166.225.170:54452->145.79.227.48:4304: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54452->145.79.227.48:4304: use of closed network connection May 13 23:49:39.534820452 (317h1m50.055644197s) [seg_pos={0 0}, upload=303679]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> May 13 23:49:39.798215243 (317h1m50.319038988s) [upload=303679]: Done uploading segments... May 13 23:49:39.798253465 (317h1m50.31907721s) [upload=303679]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.265306966s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................XX....................X] duration: 5.000947253s trace: https://jaeger.dev.storj.io/trace/791a24ba6d14f5ba FAIL (2026-05-13 23:49:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000962312s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................XX....................] duration: 3.544876885s trace: https://jaeger.dev.storj.io/trace/451a4c08eb5074da FAIL (2026-05-13 23:48:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.544901322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.....X...........XX...X..X.XX.....X.............X..X.....X..] duration: 6.61029132s trace: https://jaeger.dev.storj.io/trace/4970b937bae4c46d FAIL (2026-05-13 23:48:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.610315836s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [......X...........XX...X..X.XX.....X.............X..X.....X.] duration: 6.616716831s trace: https://jaeger.dev.storj.io/trace/4970b937bae4c46d FAIL (2026-05-13 23:48:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.616730968s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.......X...........XX...X..X.XX.....X.............X..X.....X] duration: 1m0.000849147s trace: https://jaeger.dev.storj.io/trace/4970b937bae4c46d FAIL (2026-05-13 23:48:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000871199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................XX...................] duration: 3.633243108s trace: https://jaeger.dev.storj.io/trace/451a4c08eb5074da FAIL (2026-05-13 23:47:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.633259328s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)