Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 5.000815121s trace: https://jaeger.dev.storj.io/trace/6076375f5ea6f0be FAIL (2026-01-11 17:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000843905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.829874262s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:30:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.829907044s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X.....X............XX.....X....X.........X....X..........] duration: 8.229013861s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:29:55 UTC) orbiter warning: context deadline exceeded Duration: 8.229028761s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X...XXX..X.X....XXX.......X......X...X....X.X...XX....] duration: 22.269854343s trace: https://jaeger.dev.storj.io/trace/33858a96ee5037d9 FAIL (2026-01-11 17:29:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.269875071s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [X.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 5.001462594s trace: https://jaeger.dev.storj.io/trace/6fe3887e9b3e5d5e FAIL (2026-01-11 17:29:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001487893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.727066756s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:29:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.727091533s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....X.....X............XX.....X....X.........X....X.........] duration: 21.316984082s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:29:08 UTC) orbiter warning: context deadline exceeded Duration: 21.317011283s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X......X...XXX..X.X....XXX.......X......X...X....X.X...XX...] duration: 18.081483268s trace: https://jaeger.dev.storj.io/trace/33858a96ee5037d9 FAIL (2026-01-11 17:28:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.081512894s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.001212887s trace: https://jaeger.dev.storj.io/trace/70308d61ae3f571 FAIL (2026-01-11 17:28:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001234108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.872596938s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:28:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.872616264s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X.....X............XX.....X....X.........X....X........] duration: 13.740476728s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:28:29 UTC) orbiter warning: context deadline exceeded Duration: 13.740499791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.79457987s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.7946271s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 5.001005972s trace: https://jaeger.dev.storj.io/trace/32a2d1c87d23361d FAIL (2026-01-11 17:28:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00102652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX......X...XXX..X.X....XXX.......X......X...X....X.X...XX..] duration: 19.104250469s trace: https://jaeger.dev.storj.io/trace/33858a96ee5037d9 FAIL (2026-01-11 17:27:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.104268593s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX......X...XXX..X.X....XXX.......X......X...X....X.X...XX.] duration: 14.907477351s trace: https://jaeger.dev.storj.io/trace/33858a96ee5037d9 FAIL (2026-01-11 17:27:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.907491388s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......X.....X............XX.....X....X.........X....X.......] duration: 28.78367444s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:27:29 UTC) orbiter warning: context deadline exceeded Duration: 28.78370069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [X.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 5.00144493s trace: https://jaeger.dev.storj.io/trace/3e0f9fbf6ab59335 FAIL (2026-01-11 17:27:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00146637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.759606968s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:27:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.759647447s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXX......X...XXX..X.X....XXX.......X......X...X....X.X...XX] duration: 30.001780253s trace: https://jaeger.dev.storj.io/trace/33858a96ee5037d9 FAIL (2026-01-11 17:27:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001805561s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......X.....X............XX.....X....X.........X....X......] duration: 18.951010252s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:26:38 UTC) orbiter warning: context deadline exceeded Duration: 18.951037603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.000874734s trace: https://jaeger.dev.storj.io/trace/5015209a4b983ee5 FAIL (2026-01-11 17:26:29 UTC) orbiter warning: context deadline exceeded Duration: 5.000895062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXX......X...XXX..X.X....XXX.......X......X...X....X.X...X] duration: 30.001443225s trace: https://jaeger.dev.storj.io/trace/32b8faad3bd3e94b FAIL (2026-01-11 17:26:23 UTC) orbiter warning: context deadline exceeded Duration: 30.001463854s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........X.....X............XX.....X....X.........X....X.....] duration: 35.252106255s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:26:07 UTC) orbiter warning: context deadline exceeded Duration: 35.252133096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.759304252s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:26:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.759331374s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 5.001044836s trace: https://jaeger.dev.storj.io/trace/453901b3f6eb0a20 FAIL (2026-01-11 17:25:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001075874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.709137679s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:25:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.709166945s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXX......X...XXX..X.X....XXX.......X......X...X....X.X...] duration: 22.613907659s trace: https://jaeger.dev.storj.io/trace/205fbb36ce720207 FAIL (2026-01-11 17:25:26 UTC) orbiter warning: context deadline exceeded Duration: 22.613937805s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........X.....X............XX.....X....X.........X....X....] duration: 15.514177627s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:25:19 UTC) orbiter warning: context deadline exceeded Duration: 15.514204758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.001392454s trace: https://jaeger.dev.storj.io/trace/1a91ad8ecfe39cae FAIL (2026-01-11 17:25:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0014214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.846824563s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:24:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.846853479s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........X.....X............XX.....X....X.........X....X...] duration: 25.77988415s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:24:38 UTC) orbiter warning: context deadline exceeded Duration: 25.77989943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXX......X...XXX..X.X....XXX.......X......X...X....X.X..] duration: 21.170391921s trace: https://jaeger.dev.storj.io/trace/205fbb36ce720207 FAIL (2026-01-11 17:24:33 UTC) orbiter warning: context deadline exceeded Duration: 21.17042279s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.770138915s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:24:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.770155717s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 5.000660879s trace: https://jaeger.dev.storj.io/trace/55db155f7cf751e5 FAIL (2026-01-11 17:24:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00068257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X..........X.....X............XX.....X....X.........X....X..] duration: 17.681592528s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:23:26 UTC) orbiter warning: context deadline exceeded Duration: 17.681606616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.707073715s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:23:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.707098021s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 5.001830311s trace: https://jaeger.dev.storj.io/trace/84c3f696ed22197 FAIL (2026-01-11 17:23:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001851813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....X.X.] duration: 20.643276251s trace: https://jaeger.dev.storj.io/trace/205fbb36ce720207 FAIL (2026-01-11 17:23:18 UTC) orbiter warning: context deadline exceeded Duration: 20.64329605s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....X.X] duration: 30.001321639s trace: https://jaeger.dev.storj.io/trace/205fbb36ce720207 FAIL (2026-01-11 17:22:57 UTC) orbiter warning: context deadline exceeded Duration: 30.001370652s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.788641433s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:22:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.788658247s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....X.] duration: 23.194478746s trace: https://jaeger.dev.storj.io/trace/20ce88f469855d57 FAIL (2026-01-11 17:22:17 UTC) orbiter warning: context deadline exceeded Duration: 23.194630685s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X..........X.....X............XX.....X....X.........X....X.] duration: 17.366436376s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:22:14 UTC) orbiter warning: context deadline exceeded Duration: 17.366450482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 5.001774385s trace: https://jaeger.dev.storj.io/trace/5c325d3127044d76 FAIL (2026-01-11 17:22:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001798161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X..........X.....X............XX.....X....X.........X....X] duration: 1m0.001059885s trace: https://jaeger.dev.storj.io/trace/1500fd41d26105c4 FAIL (2026-01-11 17:21:57 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001070725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.784228272s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.784255354s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....X] duration: 30.001248367s trace: https://jaeger.dev.storj.io/trace/20ce88f469855d57 FAIL (2026-01-11 17:21:51 UTC) orbiter warning: context deadline exceeded Duration: 30.001273906s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXX.XX.X.XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 5.001159233s trace: https://jaeger.dev.storj.io/trace/63adc8e38b819dca FAIL (2026-01-11 17:21:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001185091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....XXXXXXXX......X...XXX..X.X....XXX.......X......X...X....] duration: 22.973484396s trace: https://jaeger.dev.storj.io/trace/672401005a99752b FAIL (2026-01-11 17:21:13 UTC) orbiter warning: context deadline exceeded Duration: 22.973509853s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.838468008s trace: https://jaeger.dev.storj.io/trace/602790ff86ae929a FAIL (2026-01-11 17:21:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: 7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A) context deadline exceeded ======LOGS====== Jan 11 16:49:46.553445397 (889h20m46.424528452s) [upload=778256]: Starting upload Jan 11 16:49:46.553534065 (889h20m46.42461712s) [upload=778256]: Done starting upload Jan 11 16:49:46.553626842 (889h20m46.424709907s) [upload=778256]: Uploading segments... Jan 11 16:49:46.553831732 (889h20m46.424914767s) [upload=778256]: Got next segment. Inline: false Jan 11 16:49:46.553867861 (889h20m46.424950906s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 16:49:46.553873131 (889h20m46.424956176s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 16:49:47.059132086 (889h20m46.930215131s) [seg_pos={0 0}, upload=778256]: Begin upload segment... Jan 11 16:49:47.059188754 (889h20m46.930271799s) [seg_pos={0 0}, upload=778256]: Done begin upload segment. Jan 11 16:49:47.059208171 (889h20m46.930291217s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059237217 (889h20m46.930320252s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.059950645 (889h20m46.93103371s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060079601 (889h20m46.931162636s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060374893 (889h20m46.931457928s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060611824 (889h20m46.931694869s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.060840350 (889h20m46.931923385s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061166410 (889h20m46.932249445s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061404724 (889h20m46.932487759s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061638609 (889h20m46.932721644s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.061919905 (889h20m46.93300294s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062351647 (889h20m46.933434692s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062621321 (889h20m46.933704366s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.062682176 (889h20m46.933765221s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063048794 (889h20m46.934131829s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:47.063430631 (889h20m46.934513677s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Uploading piece... Jan 11 16:49:48.882124627 (889h20m48.753207672s) [upload=778256]: Next returned nil segment Jan 11 16:49:48.882140698 (889h20m48.753223733s) [upload=778256]: Waiting for error group managing segments... Jan 11 16:49:49.206883447 (889h20m49.077966492s) [noise=true , address=145.79.227.45:4229 , piece_id=FAMEBCTM5YEJEBOJOZM7RHBBBX2KNDRD5XECVBJJZKOQPQCV3NGQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.224063310 (889h20m49.095146345s) [noise=true , address=145.79.227.14:4840 , piece_id=LCJ7S3XIIUKD64ZB5SPQFTH7THUS3II4WQEU7D6XTTWNNUARCKRQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.230717120 (889h20m49.101800155s) [noise=true , address=145.79.227.50:4375 , piece_id=7MWZXII5BWZ7DXW45ZYD7YZZNRTFECQXEKJJEJNWJE366OKW6CEQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.279947370 (889h20m49.151030405s) [noise=true , address=145.79.227.49:4339 , piece_id=QXB24AZZW35WQCRMSHSQSADAHP3EC3L3FJ7OYWZWDJZ2NTNWJIYQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.291179066 (889h20m49.162262111s) [noise=true , address=145.79.227.35:4482 , piece_id=7G3WWMDPEX2J5T4EZY3JDGH44MAGHN6OOAAUSZSZ2CKAQL4JPCNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.307476138 (889h20m49.178559183s) [noise=true , address=145.79.227.2:4113 , piece_id=5QWIWYCFWXVIJTNFHCUCSCM26NLT3PT5NIUUVBSHJP5SG6MGRIJA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.310420565 (889h20m49.1815036s) [noise=true , address=145.79.227.17:4970 , piece_id=J7TNSVYZAM7U6QAMNYD4VRCAXBK65YPV67HA6IQ2HZ54N5BOSWLQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.358113939 (889h20m49.229196974s) [noise=true , address=45.91.116.12:4225 , piece_id=7ITKRJR72UXPKZSYDNVY33TELPQPCVJ5NKTFCM3H5OZ3OH6ZZDZA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.375982302 (889h20m49.247065337s) [noise=true , address=145.79.227.7:4295 , piece_id=WZS6LGGOISZ74ZY3VYOF3FZSWINH7OGPTM77TFYPQ6HJEVM2YAXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.639200633 (889h20m49.510283668s) [noise=true , address=45.91.116.7:4012 , piece_id=EWOHUJUWCNWMPGNDSE2S2AOMJAKNUTFSWN64MEO5J5L2YUOAGS4A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.701329604 (889h20m49.572412639s) [noise=true , address=45.91.116.4:4940 , piece_id=V52VPJ7V7E4YVXC4VX4V6I3KQDJ5GSCRMALCCVD3WVBSZT5U4U3Q, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:49.872769066 (889h20m49.743852111s) [noise=true , address=77.90.31.13:4626 , piece_id=Z7SNRMV2PXMTOQCBLQ2PUEFVCHVZ2IU5KS3CTAUC64PZ552R2XPQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027473682 (889h20m49.898556727s) [noise=true , address=145.79.227.11:4729 , piece_id=V3MEUVOBY6KZYJ5OFTYIRGZKYQZ5JVULKLVCKEFT22D3IRK62BXQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: <nil> Jan 11 16:49:50.027497748 (889h20m49.898580783s) [seg_pos={0 0}, upload=778256]: Segment reached optimal threshold of 13 pieces. Jan 11 16:49:50.027846753 (889h20m49.898929788s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.027858214 (889h20m49.898941249s) [noise=true , address=45.91.116.6:4976 , piece_id=GUBOKCBERCP5YIG452M4P5HCBRAHUMQ7MKLW7KYYH3NRBTNZR73A, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.027903280 (889h20m49.898986315s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48386->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 16:49:50.027911857 (889h20m49.898994892s) [noise=true , address=45.91.116.14:4190 , piece_id=NWZXCCT5CMWS3NFM3XLEMXPWVAXVRZIH75V6KGAZFIKCCFUB3BNQ, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028045792 (889h20m49.899128827s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 11 16:49:50.028054498 (889h20m49.899137533s) [noise=true , address=145.79.227.34:4905 , piece_id=IE45FOLLY73EM5FWPHNJKB43UO755CNS4WZ6X5I33WO5SXQCWJRA, seg_pos={0 0}, upload=778256]: Long tail cancellation due to optimal threshold Jan 11 16:49:50.028075528 (889h20m49.899158573s) [seg_pos={0 0}, upload=778256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 16:49:50.028158506 (889h20m49.899241551s) [upload=778256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 16:49:50.028165279 (889h20m49.899248324s) [upload=778256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 16:49:50.652255443 (889h20m50.523338489s) [upload=778256]: Done uploading segments... Jan 11 16:49:50.652275392 (889h20m50.523358427s) [upload=778256]: Upload finished. err: <nil> Duration: 5.838496622s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X..........X.....X............XX.....X....X.........X....] duration: 28.890182973s trace: https://jaeger.dev.storj.io/trace/35b8eedfa0d1f7d6 FAIL (2026-01-11 17:20:56 UTC) orbiter warning: context deadline exceeded Duration: 28.89020814s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)