Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....X.X.] duration: 20.643276251s trace: https://jaeger.dev.storj.io/trace/205fbb36ce720207 FAIL (2026-01-11 17:23:18 UTC) orbiter warning: context deadline exceeded Duration: 20.64329605s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....X.X] duration: 30.001321639s trace: https://jaeger.dev.storj.io/trace/205fbb36ce720207 FAIL (2026-01-11 17:22:57 UTC) orbiter warning: context deadline exceeded Duration: 30.001370652s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.788641433s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:22:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.788658247s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....X.] duration: 23.194478746s trace: https://jaeger.dev.storj.io/trace/20ce88f469855d57 FAIL (2026-01-11 17:22:17 UTC) orbiter warning: context deadline exceeded Duration: 23.194630685s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X..........X.....X............XX.....X....X.........X....X.] duration: 17.366436376s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:22:14 UTC) orbiter warning: context deadline exceeded Duration: 17.366450482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 5.001774385s trace: https://jaeger.dev.storj.io/trace/5c325d3127044d76 FAIL (2026-01-11 17:22:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001798161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X..........X.....X............XX.....X....X.........X....X] duration: 1m0.001059885s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:21:57 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001070725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.784228272s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.784255354s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....X] duration: 30.001248367s trace: https://jaeger.dev.storj.io/trace/20ce88f469855d57 FAIL (2026-01-11 17:21:51 UTC) orbiter warning: context deadline exceeded Duration: 30.001273906s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 5.001159233s trace: https://jaeger.dev.storj.io/trace/63adc8e38b819dca FAIL (2026-01-11 17:21:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001185091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....] duration: 22.973484396s trace: https://jaeger.dev.storj.io/trace/672401005a99752b FAIL (2026-01-11 17:21:13 UTC) orbiter warning: context deadline exceeded Duration: 22.973509853s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.838468008s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:21:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.838496622s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X..........X.....X............XX.....X....X.........X....] duration: 28.890182973s trace: https://jaeger.dev.storj.io/trace/35b8eedfa0d1f7d6 FAIL (2026-01-11 17:20:56 UTC) orbiter warning: context deadline exceeded Duration: 28.89020814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.747789634s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:20:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.747819402s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....XXXXXXXX......X...XXX..X.X....XXX.......X......X...X...] duration: 24.293333625s trace: https://jaeger.dev.storj.io/trace/672401005a99752b FAIL (2026-01-11 17:20:34 UTC) orbiter warning: context deadline exceeded Duration: 24.293371778s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X...X..........X.....X............XX.....X....X.........X...] duration: 23.248399457s trace: https://jaeger.dev.storj.io/trace/35b8eedfa0d1f7d6 FAIL (2026-01-11 17:20:18 UTC) orbiter warning: context deadline exceeded Duration: 23.24844845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 5.001605886s trace: https://jaeger.dev.storj.io/trace/427b1a03f7b95804 FAIL (2026-01-11 17:20:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001631114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.71726194s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.717290132s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.....XXXXXXXX......X...XXX..X.X....XXX.......X......X...X..] duration: 26.20657117s trace: https://jaeger.dev.storj.io/trace/672401005a99752b FAIL (2026-01-11 17:19:36 UTC) orbiter warning: context deadline exceeded Duration: 26.206595475s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X...X..........X.....X............XX.....X....X.........X..] duration: 27.519155634s trace: https://jaeger.dev.storj.io/trace/35b8eedfa0d1f7d6 FAIL (2026-01-11 17:19:27 UTC) orbiter warning: context deadline exceeded Duration: 27.519188134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 5.001120191s trace: https://jaeger.dev.storj.io/trace/4ca064ff63f50fb8 FAIL (2026-01-11 17:19:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001143676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.....XXXXXXXX......X...XXX..X.X....XXX.......X......X...X.] duration: 23.21820016s trace: https://jaeger.dev.storj.io/trace/672401005a99752b FAIL (2026-01-11 17:19:07 UTC) orbiter warning: context deadline exceeded Duration: 23.218235266s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.718437712s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:18:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.718465857s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.....XXXXXXXX......X...XXX..X.X....XXX.......X......X...X] duration: 30.001428208s trace: https://jaeger.dev.storj.io/trace/672401005a99752b FAIL (2026-01-11 17:18:44 UTC) orbiter warning: context deadline exceeded Duration: 30.001466111s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 5.001457881s trace: https://jaeger.dev.storj.io/trace/439a8d776bc9435c FAIL (2026-01-11 17:18:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00151481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.769473696s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:18:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.76949728s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X.....XXXXXXXX......X...XXX..X.X....XXX.......X......X...] duration: 23.436003663s trace: https://jaeger.dev.storj.io/trace/5734ea9e5fea046d FAIL (2026-01-11 17:18:14 UTC) orbiter warning: context deadline exceeded Duration: 23.436025174s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X...X..........X.....X............XX.....X....X.........X.] duration: 22.972602221s trace: https://jaeger.dev.storj.io/trace/35b8eedfa0d1f7d6 FAIL (2026-01-11 17:18:14 UTC) orbiter warning: context deadline exceeded Duration: 22.97262766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X...X..........X.....X............XX.....X....X.........X] duration: 1m0.001407857s trace: https://jaeger.dev.storj.io/trace/35b8eedfa0d1f7d6 FAIL (2026-01-11 17:17:51 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001426173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.758442783s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:17:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.758468815s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.001030136s trace: https://jaeger.dev.storj.io/trace/4aca8a86fc198901 FAIL (2026-01-11 17:17:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001102465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX.X.....XXXXXXXX......X...XXX..X.X....XXX.......X......X..] duration: 23.95388989s trace: https://jaeger.dev.storj.io/trace/5734ea9e5fea046d FAIL (2026-01-11 17:17:21 UTC) orbiter warning: context deadline exceeded Duration: 23.953905069s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.] duration: 5.001159108s trace: https://jaeger.dev.storj.io/trace/4e966bed156b0146 FAIL (2026-01-11 17:17:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00118576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XX.X.....XXXXXXXX......X...XXX..X.X....XXX.......X......X.] duration: 24.064080344s trace: https://jaeger.dev.storj.io/trace/5734ea9e5fea046d FAIL (2026-01-11 17:16:57 UTC) orbiter warning: context deadline exceeded Duration: 24.064097568s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.740052605s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:16:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.740089024s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002342595s trace: https://jaeger.dev.storj.io/trace/4e966bed156b0146 FAIL (2026-01-11 17:16:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002370829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.XX.X.....XXXXXXXX......X...XXX..X.X....XXX.......X......X] duration: 30.001724297s trace: https://jaeger.dev.storj.io/trace/5734ea9e5fea046d FAIL (2026-01-11 17:16:33 UTC) orbiter warning: context deadline exceeded Duration: 30.00175266s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000871022s trace: https://jaeger.dev.storj.io/trace/66ba19800f698668 FAIL (2026-01-11 17:16:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000895879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....X...X..........X.....X............XX.....X....X.........] duration: 19.213328685s trace: https://jaeger.dev.storj.io/trace/32f3c818bdbcb6c2 FAIL (2026-01-11 17:15:55 UTC) orbiter warning: context deadline exceeded Duration: 19.213352199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X.XX.X.....XXXXXXXX......X...XXX..X.X....XXX.......X......] duration: 21.874537526s trace: https://jaeger.dev.storj.io/trace/2a96b479bc8106a1 FAIL (2026-01-11 17:15:44 UTC) orbiter warning: context deadline exceeded Duration: 21.874552244s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 5.001129067s trace: https://jaeger.dev.storj.io/trace/708a5f2e11345d75 FAIL (2026-01-11 17:15:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001152923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.767754412s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:15:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.767781484s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X.XX.X.....XXXXXXXX......X...XXX..X.X....XXX.......X.....] duration: 20.796975796s trace: https://jaeger.dev.storj.io/trace/2a96b479bc8106a1 FAIL (2026-01-11 17:15:22 UTC) orbiter warning: context deadline exceeded Duration: 20.797051859s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X...X..........X.....X............XX.....X....X........] duration: 20.764902408s trace: https://jaeger.dev.storj.io/trace/32f3c818bdbcb6c2 FAIL (2026-01-11 17:15:12 UTC) orbiter warning: context deadline exceeded Duration: 20.764928175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 5.000861615s trace: https://jaeger.dev.storj.io/trace/59bed3422791041a FAIL (2026-01-11 17:15:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000889448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.690456562s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:14:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.690486388s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 5.000945836s trace: https://jaeger.dev.storj.io/trace/4f34be19ab2ba8be FAIL (2026-01-11 17:14:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000969731s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X.X.XX.X.....XXXXXXXX......X...XXX..X.X....XXX.......X....] duration: 29.341016763s trace: https://jaeger.dev.storj.io/trace/2a96b479bc8106a1 FAIL (2026-01-11 17:14:17 UTC) orbiter warning: context deadline exceeded Duration: 29.341049797s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.939401933s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.939454672s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 5.00074819s trace: https://jaeger.dev.storj.io/trace/5c0159b481a2a245 FAIL (2026-01-11 17:13:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000773909s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)