Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.86295376s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.862981371s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....XXX..X.X.] duration: 30.000600372s trace: https://jaeger.dev.storj.io/trace/16cc2b32e74b441a FAIL (2026-01-13 15:50:34 UTC) orbiter warning: context deadline exceeded Duration: 30.000613255s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X.........X...X........X......X....XX.X.................X...] duration: 19.806026752s trace: https://jaeger.dev.storj.io/trace/6148b8276b96a1e3 FAIL (2026-01-13 15:50:30 UTC) orbiter warning: context deadline exceeded Duration: 19.806734579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001881737s trace: https://jaeger.dev.storj.io/trace/1599f7503b598e83 FAIL (2026-01-13 15:50:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001907525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XX.........X...X........X......X....XX.X.................X..] duration: 20.550511662s trace: https://jaeger.dev.storj.io/trace/6148b8276b96a1e3 FAIL (2026-01-13 15:50:08 UTC) orbiter warning: context deadline exceeded Duration: 20.550537722s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....XXX..X.X] duration: 30.001941644s trace: https://jaeger.dev.storj.io/trace/16cc2b32e74b441a FAIL (2026-01-13 15:50:04 UTC) orbiter warning: context deadline exceeded Duration: 30.001967113s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000807409s trace: https://jaeger.dev.storj.io/trace/530d2e04e64b0929 FAIL (2026-01-13 15:49:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000831366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.784518136s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:49:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.784550246s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001708477s trace: https://jaeger.dev.storj.io/trace/43b488029f4fcc18 FAIL (2026-01-13 15:49:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001731129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 15.164212977s trace: https://jaeger.dev.storj.io/trace/7714c22d9ac209f0 FAIL (2026-01-13 15:49:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 15.164241029s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....XXX..X.] duration: 28.877541044s trace: https://jaeger.dev.storj.io/trace/1d6be617d35c431c FAIL (2026-01-13 15:49:30 UTC) orbiter warning: context deadline exceeded Duration: 28.877557406s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.580593991s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:49:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.580620191s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.XX.........X...X........X......X....XX.X.................X.] duration: 17.959814527s trace: https://jaeger.dev.storj.io/trace/6148b8276b96a1e3 FAIL (2026-01-13 15:49:08 UTC) orbiter warning: context deadline exceeded Duration: 17.959840096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.64364599s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:49:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.643667591s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....XXX..X] duration: 30.00098588s trace: https://jaeger.dev.storj.io/trace/1d6be617d35c431c FAIL (2026-01-13 15:49:01 UTC) orbiter warning: context deadline exceeded Duration: 30.001021598s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..XX.........X...X........X......X....XX.X.................X] duration: 1m0.001378366s trace: https://jaeger.dev.storj.io/trace/6148b8276b96a1e3 FAIL (2026-01-13 15:48:42 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001402182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.670646982s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:48:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.670674073s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 14.686336887s trace: https://jaeger.dev.storj.io/trace/7714c22d9ac209f0 FAIL (2026-01-13 15:48:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 14.686361293s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001624906s trace: https://jaeger.dev.storj.io/trace/7a2b52881b35f66e FAIL (2026-01-13 15:48:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001646235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....XXX..] duration: 22.400544904s trace: https://jaeger.dev.storj.io/trace/46fba89668603902 FAIL (2026-01-13 15:48:18 UTC) orbiter warning: context deadline exceeded Duration: 22.400573389s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 8.78062555s trace: https://jaeger.dev.storj.io/trace/7714c22d9ac209f0 FAIL (2026-01-13 15:48:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.780650006s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000747926s trace: https://jaeger.dev.storj.io/trace/662a09706e0e4dd7 FAIL (2026-01-13 15:47:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000787953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....XXX.] duration: 29.931401402s trace: https://jaeger.dev.storj.io/trace/46fba89668603902 FAIL (2026-01-13 15:47:53 UTC) orbiter warning: context deadline exceeded Duration: 29.931427062s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.925892072s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.925918602s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...XX.........X...X........X......X....XX.X.................] duration: 27.360242493s trace: https://jaeger.dev.storj.io/trace/11c66761863eeec0 FAIL (2026-01-13 15:47:28 UTC) orbiter warning: context deadline exceeded Duration: 27.360265215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.619816199s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:47:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.619842138s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 25.539881657s trace: https://jaeger.dev.storj.io/trace/7714c22d9ac209f0 FAIL (2026-01-13 15:47:17 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 25.539907056s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....XXX] duration: 30.000857061s trace: https://jaeger.dev.storj.io/trace/46fba89668603902 FAIL (2026-01-13 15:47:02 UTC) orbiter warning: context deadline exceeded Duration: 30.00093461s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001192853s trace: https://jaeger.dev.storj.io/trace/5713a84c59a2272d FAIL (2026-01-13 15:47:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001215596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.690531745s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:46:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.690572092s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.77988579s trace: https://jaeger.dev.storj.io/trace/7714c22d9ac209f0 FAIL (2026-01-13 15:46:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.77990615s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001060902s trace: https://jaeger.dev.storj.io/trace/5bd3852852ede7cf FAIL (2026-01-13 15:46:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001079579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....XX.........X...X........X......X....XX.X................] duration: 34.237923337s trace: https://jaeger.dev.storj.io/trace/11c66761863eeec0 FAIL (2026-01-13 15:46:25 UTC) orbiter warning: context deadline exceeded Duration: 34.237950237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XX..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....XX] duration: 30.001484249s trace: https://jaeger.dev.storj.io/trace/5ce961f130b9fd2f FAIL (2026-01-13 15:46:18 UTC) orbiter warning: context deadline exceeded Duration: 30.001505148s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.642026557s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:46:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.642060441s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 20.639599818s trace: https://jaeger.dev.storj.io/trace/7714c22d9ac209f0 FAIL (2026-01-13 15:46:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 20.639622251s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....XX.........X...X........X......X....XX.X...............] duration: 23.586196699s trace: https://jaeger.dev.storj.io/trace/11c66761863eeec0 FAIL (2026-01-13 15:45:18 UTC) orbiter warning: context deadline exceeded Duration: 23.586221425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001156496s trace: https://jaeger.dev.storj.io/trace/9cf30174b3b5fa0 FAIL (2026-01-13 15:45:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001181062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.764581793s trace: https://jaeger.dev.storj.io/trace/7714c22d9ac209f0 FAIL (2026-01-13 15:45:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.764612432s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...XX..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....X] duration: 30.001506253s trace: https://jaeger.dev.storj.io/trace/6f1ce97343357167 FAIL (2026-01-13 15:45:09 UTC) orbiter warning: context deadline exceeded Duration: 30.001530148s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.663735856s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:45:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.663764411s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X...XX..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX....] duration: 29.831363593s trace: https://jaeger.dev.storj.io/trace/541d1268be2534a2 FAIL (2026-01-13 15:44:39 UTC) orbiter warning: context deadline exceeded Duration: 29.831434067s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [......XX.........X...X........X......X....XX.X..............] duration: 30.544544595s trace: https://jaeger.dev.storj.io/trace/11c66761863eeec0 FAIL (2026-01-13 15:44:31 UTC) orbiter warning: context deadline exceeded Duration: 30.544560164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.82568254s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:44:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.825755057s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000740131s trace: https://jaeger.dev.storj.io/trace/40230867f4e11b04 FAIL (2026-01-13 15:44:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000768765s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X...XX..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX...] duration: 20.611119282s trace: https://jaeger.dev.storj.io/trace/541d1268be2534a2 FAIL (2026-01-13 15:44:05 UTC) orbiter warning: context deadline exceeded Duration: 20.611135143s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.673557165s trace: https://jaeger.dev.storj.io/trace/7714c22d9ac209f0 FAIL (2026-01-13 15:44:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.673587694s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.......XX.........X...X........X......X....XX.X.............] duration: 55.186031619s trace: https://jaeger.dev.storj.io/trace/11c66761863eeec0 FAIL (2026-01-13 15:44:00 UTC) orbiter warning: context deadline exceeded Duration: 55.186046969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X...XX..XXXXXXXXX..X...X..X.XX....X.X.X..X..X.......XXXX..] duration: 26.336895724s trace: https://jaeger.dev.storj.io/trace/541d1268be2534a2 FAIL (2026-01-13 15:43:44 UTC) orbiter warning: context deadline exceeded Duration: 26.336918167s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.635568933s trace: https://jaeger.dev.storj.io/trace/7e7ccabac934ab30 FAIL (2026-01-13 15:43:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 13 15:19:53.520465654 (935h50m53.391548699s) [upload=819113]: Starting upload Jan 13 15:19:53.520565765 (935h50m53.3916488s) [upload=819113]: Done starting upload Jan 13 15:19:53.520776226 (935h50m53.391859261s) [upload=819113]: Uploading segments... Jan 13 15:19:53.520930850 (935h50m53.392013885s) [upload=819113]: Got next segment. Inline: false Jan 13 15:19:53.520972348 (935h50m53.392055393s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 15:19:53.520977619 (935h50m53.392060664s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 15:19:53.787982748 (935h50m53.659065783s) [seg_pos={0 0}, upload=819113]: Begin upload segment... Jan 13 15:19:53.788025619 (935h50m53.659108654s) [seg_pos={0 0}, upload=819113]: Done begin upload segment. Jan 13 15:19:53.788049956 (935h50m53.659132991s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788234878 (935h50m53.659317923s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.788927617 (935h50m53.660010662s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789438991 (935h50m53.660522026s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.789953290 (935h50m53.661036325s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790499750 (935h50m53.661582786s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.790829568 (935h50m53.661912603s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.791432006 (935h50m53.662515061s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792110938 (935h50m53.663193983s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.792979793 (935h50m53.664062828s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793683883 (935h50m53.664766928s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.793984867 (935h50m53.665067902s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794170289 (935h50m53.665253324s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794491080 (935h50m53.665574125s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.794641586 (935h50m53.665724631s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:53.795011481 (935h50m53.666094516s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Uploading piece... Jan 13 15:19:55.644030328 (935h50m55.515113373s) [upload=819113]: Next returned nil segment Jan 13 15:19:55.644049335 (935h50m55.51513237s) [upload=819113]: Waiting for error group managing segments... Jan 13 15:19:55.945093979 (935h50m55.816177014s) [noise=true , address=145.79.227.3:4151 , piece_id=4YHKIBFMS4F2NZJT2RSBH6YK6TVZPXB4GHYUVMBLQ2VGUBFXIIGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.948471530 (935h50m55.819554575s) [noise=true , address=145.79.227.9:4365 , piece_id=6HDHSB2GIVUL4XEQIEZ4RM6V75PUSRNYTULQGQZSBKLXHQVOWSGQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.971222189 (935h50m55.842305234s) [noise=true , address=145.79.227.21:4474 , piece_id=FEZLZW3N7RWFEJUVD4U4CYQH36EA42S6575ABYZIKMGJEDKFJO6A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.972821975 (935h50m55.84390501s) [noise=true , address=145.79.227.38:4979 , piece_id=OQXGSKK2XPK4SIUQBXSPPJRUNJWRMRMPMLPYJIICQZIYVDKP47WA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997579746 (935h50m55.868662781s) [noise=true , address=145.79.227.17:4970 , piece_id=SKQB2WCUGRWID6ERV2VVQJPLHY6AISFWPOOZHGLUUVCC65HBF7GA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:55.997617578 (935h50m55.868700623s) [noise=true , address=145.79.227.27:4690 , piece_id=DVZ56N3VMLMFWWK4JWO57MHY75DQXJZOY57SCNNHGB6T73QKHDYA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.028405250 (935h50m55.899488286s) [noise=true , address=145.79.227.42:4122 , piece_id=QINS3LAR6SVZOUOCDCM3XXZGF7ZRBSIQZMLJLW6A6WXNWB2V54WQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.034773927 (935h50m55.905856972s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZ4TCXB2E56J36ZN6ULDEQVS5FYIKHL3UMGWILXYHTPIR6E567VQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.040883119 (935h50m55.911966164s) [noise=true , address=145.79.227.41:4086 , piece_id=5YIJSDL2HLKE3BDCXDP5K6XB56ERK4JBZAMSZSIAKWQC53D5SY5Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.057242710 (935h50m55.928325745s) [noise=true , address=145.79.227.51:4410 , piece_id=NY3MHUBFOOTFNYGYZR4OPRRWQDJ5C7H6PQAXYUSC2JFABKO63HWA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.070497577 (935h50m55.941580612s) [noise=true , address=45.91.116.8:4045 , piece_id=G5HFZ42CDKCTBCV4PGPHN7VP4XBEE2A5474UP3QPSMK5BRUFSVRA, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.346494328 (935h50m56.217577363s) [noise=true , address=145.79.227.36:4519 , piece_id=MEQPZ2E3EH765P3TWSFTEJTOENEGMRLZ7I72NI37CWLFBJS7X4CQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766602729 (935h50m56.637685764s) [noise=true , address=145.79.227.18:4007 , piece_id=QYIQJYIAI6MF3YOMLDOQEFO56XEHZG65FXQ5QKCB7Z7BY2FPSKVQ, seg_pos={0 0}, upload=819113]: Done uploading piece. err: <nil> Jan 13 15:19:56.766625623 (935h50m56.637708658s) [seg_pos={0 0}, upload=819113]: Segment reached optimal threshold of 13 pieces. Jan 13 15:19:56.766839680 (935h50m56.637922725s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.766847906 (935h50m56.637930941s) [noise=true , address=145.79.227.7:4295 , piece_id=2NJXZWEREYNI3FYWBKMFHRTRDDVLCMUPCEBPCRDC2R6KRXBZ5L4A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767169188 (935h50m56.638252223s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767193243 (935h50m56.638276279s) [noise=true , address=45.91.116.9:4264 , piece_id=QLBMB5OSNYNUKZ35Y6BFI2VHYAPI5NXQAKUOTOON3YD3BO5PW45A, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767338090 (935h50m56.638421125s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jan 13 15:19:56.767347408 (935h50m56.638430443s) [noise=true , address=45.91.116.12:4226 , piece_id=73J77DJKZE2F56VBLP3FGRUUA6Z4KE7HXXLPRNFKMULU42TKG62Q, seg_pos={0 0}, upload=819113]: Long tail cancellation due to optimal threshold Jan 13 15:19:56.767371122 (935h50m56.638454167s) [seg_pos={0 0}, upload=819113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 15:19:56.767451887 (935h50m56.638534932s) [upload=819113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 15:19:56.767456645 (935h50m56.638539681s) [upload=819113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 15:19:57.077205824 (935h50m56.948288869s) [upload=819113]: Done uploading segments... Jan 13 15:19:57.077228487 (935h50m56.948311532s) [upload=819113]: Upload finished. err: <nil> Duration: 5.63559919s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)