Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001581905s trace: https://jaeger.dev.storj.io/trace/6f5ed270c44d977c FAIL (2026-01-09 06:21:49 UTC) orbiter warning: context deadline exceeded Duration: 5.001603537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 22.425511007s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:21:46 UTC) orbiter warning: context deadline exceeded Duration: 22.425533732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........XX....X...........X........X.X...XX.X..X.........] duration: 6.171738179s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 6.171761774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............XX....X...........X........X.X...XX.X..X........] duration: 6.001662181s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:21:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 6.001689082s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 16.283841002s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:21:02 UTC) orbiter warning: context deadline exceeded Duration: 16.28386659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000998143s trace: https://jaeger.dev.storj.io/trace/38edff0a6cb97ba2 FAIL (2026-01-09 06:20:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001021257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 15.949153009s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:20:23 UTC) orbiter warning: context deadline exceeded Duration: 15.949178999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001552271s trace: https://jaeger.dev.storj.io/trace/31cea0a66b6dbc60 FAIL (2026-01-09 06:20:18 UTC) orbiter warning: context deadline exceeded Duration: 5.001575494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............XX....X...........X........X.X...XX.X..X.......] duration: 6.088835443s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:19:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 6.088866703s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001711066s trace: https://jaeger.dev.storj.io/trace/683bd481511010bd FAIL (2026-01-09 06:19:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001733689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 15.444543048s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:19:13 UTC) orbiter warning: context deadline exceeded Duration: 15.444571793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............XX....X...........X........X.X...XX.X..X......] duration: 5.903419033s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:19:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 5.903442038s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 23.147368007s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:18:48 UTC) orbiter warning: context deadline exceeded Duration: 23.147392462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001079547s trace: https://jaeger.dev.storj.io/trace/40f71e3dd7f2ed0b FAIL (2026-01-09 06:18:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001357027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...............XX....X...........X........X.X...XX.X..X.....] duration: 5.869435335s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:18:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 5.869466303s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000825571s trace: https://jaeger.dev.storj.io/trace/55a3efb82310da8e FAIL (2026-01-09 06:18:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000859918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 16.802250631s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:17:56 UTC) orbiter warning: context deadline exceeded Duration: 16.802293062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [................XX....X...........X........X.X...XX.X..X....] duration: 6.109704006s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:17:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 6.109731819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................XX....X...........X........X.X...XX.X..X...] duration: 5.854384313s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:17:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 5.854415161s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001015377s trace: https://jaeger.dev.storj.io/trace/75b982b7f5297bfc FAIL (2026-01-09 06:17:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001035877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 20.232206862s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:17:00 UTC) orbiter warning: context deadline exceeded Duration: 20.232230316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000715343s trace: https://jaeger.dev.storj.io/trace/702d15f249616ccc FAIL (2026-01-09 06:16:31 UTC) orbiter warning: context deadline exceeded Duration: 5.000731053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................XX....X...........X........X.X...XX.X..X..] duration: 5.765266229s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:16:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 5.765297088s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [.XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001079601s trace: https://jaeger.dev.storj.io/trace/93b28508d166452 FAIL (2026-01-09 06:16:04 UTC) orbiter warning: context deadline exceeded Duration: 5.001102543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 17.25215284s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:16:04 UTC) orbiter warning: context deadline exceeded Duration: 17.252177566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..................XX....X...........X........X.X...XX.X..X.] duration: 6.135325657s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:15:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 6.135354113s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [X.XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000876618s trace: https://jaeger.dev.storj.io/trace/333e68fa038e1ef3 FAIL (2026-01-09 06:15:38 UTC) orbiter warning: context deadline exceeded Duration: 5.000886867s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.997742101s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:15:38 UTC) orbiter warning: context deadline exceeded Duration: 15.997767338s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 20.701923345s trace: https://jaeger.dev.storj.io/trace/2f53d28901977662 FAIL (2026-01-09 06:15:24 UTC) orbiter warning: context deadline exceeded Duration: 20.701956208s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 12.815561908s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:15:15 UTC) orbiter warning: context deadline exceeded Duration: 12.815598095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..................XX....X...........X........X.X...XX.X..X] duration: 30.002450754s trace: https://jaeger.dev.storj.io/trace/767dc81c2be0328 FAIL (2026-01-09 06:15:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A) context deadline exceeded ======LOGS====== 9 06:14:44.133209750 (830h45m44.004292795s) [upload=726663]: Starting upload Jan 9 06:14:44.133266649 (830h45m44.004349684s) [upload=726663]: Done starting upload Jan 9 06:14:44.133397809 (830h45m44.004480844s) [upload=726663]: Uploading segments... Jan 9 06:14:44.133499823 (830h45m44.004582868s) [upload=726663]: Got next segment. Inline: false Jan 9 06:14:44.133570127 (830h45m44.004653162s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:14:44.133577421 (830h45m44.004660456s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:14:44.376950314 (830h45m44.248033349s) [seg_pos={0 0}, upload=726663]: Begin upload segment... Jan 9 06:14:44.377003566 (830h45m44.248086601s) [seg_pos={0 0}, upload=726663]: Done begin upload segment. Jan 9 06:14:44.377040186 (830h45m44.248123231s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377085161 (830h45m44.248168206s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377548564 (830h45m44.248631609s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.377722315 (830h45m44.24880536s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378124310 (830h45m44.249207355s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378312648 (830h45m44.249395683s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.378696029 (830h45m44.249779064s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379067125 (830h45m44.25015016s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379131979 (830h45m44.250215024s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379615760 (830h45m44.250698805s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.379960707 (830h45m44.251043742s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380240650 (830h45m44.251323695s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380602809 (830h45m44.251685844s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.380827668 (830h45m44.251910703s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381186972 (830h45m44.252270007s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:44.381350423 (830h45m44.252433459s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Uploading piece... Jan 9 06:14:46.084711067 (830h45m45.955794102s) [upload=726663]: Next returned nil segment Jan 9 06:14:46.084724584 (830h45m45.955807619s) [upload=726663]: Waiting for error group managing segments... Jan 9 06:14:46.371445789 (830h45m46.242528824s) [noise=true , address=145.79.227.40:4051 , piece_id=EX7L7YFMYVFVOD7HXLGYREYPKYO65PJD2TR3WNPW336KFGU4FG2A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.372939333 (830h45m46.244022378s) [noise=true , address=145.79.227.46:4601 , piece_id=QNSFDJC7AQ6673AWTSZKY5UL3D27LVY5GORBAKTCOQWC7C3VTOXA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385195200 (830h45m46.256278235s) [noise=true , address=145.79.227.48:4303 , piece_id=OCWVSWRKTUAXAXIIRKMUNBP3TMKZRWZXQ75A5PR7F7P7ZAGOKHVQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.385700692 (830h45m46.256783737s) [noise=true , address=145.79.227.5:4223 , piece_id=HKNTX2GPGGLV3OJSJX4SF7CUZJEXNBC6UG7H5SRS32KA6XKF7WOA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.390346538 (830h45m46.261429573s) [noise=true , address=145.79.227.23:4545 , piece_id=DWVKREBQV3DHLPMYGHP42TC7Q7SIBUYXK47K7RMP4KME5LAZFW7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.398302218 (830h45m46.269385263s) [noise=true , address=145.79.227.6:4258 , piece_id=XMBTQWFHLLQA7HGVKNFXUBSHGUSTKONDEBDX3BPZMTBRTQFG6OXQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.418777034 (830h45m46.289860069s) [noise=true , address=145.79.227.44:4193 , piece_id=6RBDB2YZHV5DBUA2NNT7JVSK3DUJWHSEVEBJILEFD7QLSRXGHVYQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.445630346 (830h45m46.316713381s) [noise=true , address=145.79.227.50:4375 , piece_id=AAV6EDI54SAZD4EAKVWINWHWNSS2LIPADOPF5YI2ENCHJYZEX4PQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.554868320 (830h45m46.425951355s) [noise=true , address=145.79.227.41:4085 , piece_id=6EJRRATJ2JIPQL3MBL63LLGIMJKEU2ZFE2KZZ4J3IN5USFD2VXMQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.735038076 (830h45m46.606121121s) [noise=true , address=77.90.31.12:4589 , piece_id=OJBODWX7P42G2JV4XDNVLWABJV65DTWYSN56CBSGSKDLZJXUQG3A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.752429242 (830h45m46.623512287s) [noise=true , address=45.91.116.11:4119 , piece_id=CNNARZRIRCWAEUPIKBUHUM5GPHLYB7S6IT3WJKX5DMGCM4BDPC7Q, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.777515640 (830h45m46.648598685s) [noise=true , address=45.91.116.4:4940 , piece_id=RDWEILV4CRLQQLQCVTDFHZVNGU3R4OBPQ2GTZBQ73U7MRJAR6FCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780170044 (830h45m46.651253089s) [noise=true , address=45.91.116.13:4154 , piece_id=WLC3GPXZ7BQ3NHQEQFE6L62BTAT4OJJZXI66FUAFUN2PYHT3ASUQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: <nil> Jan 9 06:14:46.780182227 (830h45m46.651265272s) [seg_pos={0 0}, upload=726663]: Segment reached optimal threshold of 13 pieces. Jan 9 06:14:46.780328837 (830h45m46.651411872s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780336381 (830h45m46.651419416s) [noise=true , address=77.90.31.17:4770 , piece_id=RKRLXGGUZ4BYQXSRBK2VB3P7IXJC3J7S2EBECWKOVEL77IN7YLCQ, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780487759 (830h45m46.651570804s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Jan 9 06:14:46.780498510 (830h45m46.651581545s) [noise=true , address=45.91.116.5:4299 , piece_id=PCBBNMMNMYWX4XZ2252IMP5QII3LZ777GNBOJPRIYMNDM2VCV46A, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780537825 (830h45m46.65162086s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59572->77.90.31.18:4805: use of closed network connection Jan 9 06:14:46.780544738 (830h45m46.651627773s) [noise=true , address=77.90.31.18:4805 , piece_id=DEOZOWTLXMIFSTBA74EWVXN7SQLDZVKFA5XB572VHGLCY4UOWVCA, seg_pos={0 0}, upload=726663]: Long tail cancellation due to optimal threshold Jan 9 06:14:46.780559826 (830h45m46.651642861s) [seg_pos={0 0}, upload=726663]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:14:46.780622375 (830h45m46.65170541s) [upload=726663]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:14:46.780626643 (830h45m46.651709688s) [upload=726663]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:14:47.082429286 (830h45m46.953512321s) [upload=726663]: Done uploading segments... Jan 9 06:14:47.082454053 (830h45m46.953537098s) [upload=726663]: Upload finished. err: <nil> Duration: 30.002483646s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XX.XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.005818308s trace: https://jaeger.dev.storj.io/trace/1e0044733fc248df FAIL (2026-01-09 06:15:01 UTC) orbiter warning: context deadline exceeded Duration: 5.005845228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 16.97609524s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:15:01 UTC) orbiter warning: context deadline exceeded Duration: 16.97612199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 7.201922666s trace: https://jaeger.dev.storj.io/trace/2f53d28901977662 FAIL (2026-01-09 06:14:53 UTC) orbiter warning: context deadline exceeded Duration: 7.201953043s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.36569412s trace: https://jaeger.dev.storj.io/trace/2f53d28901977662 FAIL (2026-01-09 06:14:34 UTC) orbiter warning: context deadline exceeded Duration: 7.36571981s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXX.XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001361455s trace: https://jaeger.dev.storj.io/trace/75c91081afbf6070 FAIL (2026-01-09 06:14:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001387323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 17.275371656s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:14:02 UTC) orbiter warning: context deadline exceeded Duration: 17.275392767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X..................XX....X...........X........X.X...XX.X..] duration: 6.13619567s trace: https://jaeger.dev.storj.io/trace/592dc21013563f8f FAIL (2026-01-09 06:13:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q) context deadline exceeded ======LOGS====== Jan 9 06:12:47.163506303 (830h43m47.034589348s) [upload=726638]: Starting upload Jan 9 06:12:47.163564785 (830h43m47.03464782s) [upload=726638]: Done starting upload Jan 9 06:12:47.163741030 (830h43m47.034824065s) [upload=726638]: Uploading segments... Jan 9 06:12:47.163774524 (830h43m47.034857569s) [upload=726638]: Got next segment. Inline: false Jan 9 06:12:47.163799531 (830h43m47.034882576s) [upload=726638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:12:47.163804091 (830h43m47.034887126s) [upload=726638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:12:47.408937297 (830h43m47.280020332s) [seg_pos={0 0}, upload=726638]: Begin upload segment... Jan 9 06:12:47.408990427 (830h43m47.280073472s) [seg_pos={0 0}, upload=726638]: Done begin upload segment. Jan 9 06:12:47.409020505 (830h43m47.28010358s) [noise=true , address=145.79.227.23:4545 , piece_id=IZYOX3RMPU6GWWATG2LF4FGY6ORHOTV2SEYCK4CRNPIJIN5CDYCQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.409077524 (830h43m47.280160569s) [noise=true , address=145.79.227.3:4150 , piece_id=OOBMM5FL2JMFANGWZBRIXCIYBWJAJ3VP47PFCDVN3UTJHAYCYJBQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.410106573 (830h43m47.281189618s) [noise=true , address=45.91.116.5:4300 , piece_id=2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.410734779 (830h43m47.281817824s) [noise=true , address=145.79.227.42:4121 , piece_id=U52S7JVFLTKSEOA4JKHL4AGQ45SQZHW577WRRYGKXQO7H4DGKWZQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411197601 (830h43m47.282280646s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411688235 (830h43m47.28277128s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411917261 (830h43m47.283000296s) [noise=true , address=145.79.227.34:4905 , piece_id=I5O23IEXQYMJPVHNXH5YSGFP3ZAZZUZNWW7TOIZMWOASTPO6CJAQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412162217 (830h43m47.283245262s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412450186 (830h43m47.283533221s) [noise=true , address=45.91.116.8:4047 , piece_id=IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412592587 (830h43m47.283675632s) [noise=true , address=145.79.227.48:4303 , piece_id=OQBE6662QB447LECLADWKKFE573PF6OGJIJOOSSIE7CCSJNHWIMQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412764935 (830h43m47.28384798s) [noise=true , address=145.79.227.44:4193 , piece_id=HO3HEQQVVE5LBETZSEYQXG4GL5O33S6XDU4WDDHPYDXVSP7PDR7Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413020271 (830h43m47.284103316s) [noise=true , address=145.79.227.49:4337 , piece_id=I37AARED2Z2GGJ2FOG2K3425CCIEKN5EEFSPIYKTFUBPPZS73L6Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413124560 (830h43m47.284207605s) [noise=true , address=145.79.227.9:4367 , piece_id=QJXFC7CALWQ2IEGPKFLT6QGNYYPJ75BNKGANVICKG4VHV7ACUDPA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413464157 (830h43m47.284547202s) [noise=true , address=145.79.227.36:4519 , piece_id=F5SEZVZ3I2MW6XIM463IXSGKYCBIHHJE2XN4JNZ56BQZSL2VKW2Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.414009855 (830h43m47.28509289s) [noise=true , address=145.79.227.46:4601 , piece_id=TO2LWDLQH7HBTNKDT4IQXIIUVRHE4LE4GGIUKPCN4A6QSU7RQFIQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.414503566 (830h43m47.285586611s) [noise=true , address=145.79.227.38:4978 , piece_id=JGA3CCDJPAA6OKIT7IPQO25OQMK47OOCJDOMKP2TWN73WDE6F2AQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:49.005318296 (830h43m48.876401351s) [upload=726638]: Next returned nil segment Jan 9 06:12:49.005332112 (830h43m48.876415157s) [upload=726638]: Waiting for error group managing segments... Jan 9 06:12:49.386947948 (830h43m49.258030983s) [noise=true , address=145.79.227.36:4519 , piece_id=F5SEZVZ3I2MW6XIM463IXSGKYCBIHHJE2XN4JNZ56BQZSL2VKW2Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.397771488 (830h43m49.268854523s) [noise=true , address=145.79.227.3:4150 , piece_id=OOBMM5FL2JMFANGWZBRIXCIYBWJAJ3VP47PFCDVN3UTJHAYCYJBQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.399566335 (830h43m49.27064937s) [noise=true , address=145.79.227.49:4337 , piece_id=I37AARED2Z2GGJ2FOG2K3425CCIEKN5EEFSPIYKTFUBPPZS73L6Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.402655958 (830h43m49.273738993s) [noise=true , address=145.79.227.23:4545 , piece_id=IZYOX3RMPU6GWWATG2LF4FGY6ORHOTV2SEYCK4CRNPIJIN5CDYCQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.405514451 (830h43m49.276597496s) [noise=true , address=145.79.227.44:4193 , piece_id=HO3HEQQVVE5LBETZSEYQXG4GL5O33S6XDU4WDDHPYDXVSP7PDR7Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.415164376 (830h43m49.286247421s) [noise=true , address=145.79.227.34:4905 , piece_id=I5O23IEXQYMJPVHNXH5YSGFP3ZAZZUZNWW7TOIZMWOASTPO6CJAQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.416915090 (830h43m49.287998125s) [noise=true , address=145.79.227.48:4303 , piece_id=OQBE6662QB447LECLADWKKFE573PF6OGJIJOOSSIE7CCSJNHWIMQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.418002189 (830h43m49.289085224s) [noise=true , address=145.79.227.42:4121 , piece_id=U52S7JVFLTKSEOA4JKHL4AGQ45SQZHW577WRRYGKXQO7H4DGKWZQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.422796048 (830h43m49.293879083s) [noise=true , address=145.79.227.38:4978 , piece_id=JGA3CCDJPAA6OKIT7IPQO25OQMK47OOCJDOMKP2TWN73WDE6F2AQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.485495085 (830h43m49.35657812s) [noise=true , address=145.79.227.9:4367 , piece_id=QJXFC7CALWQ2IEGPKFLT6QGNYYPJ75BNKGANVICKG4VHV7ACUDPA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.486332329 (830h43m49.357415364s) [noise=true , address=145.79.227.46:4601 , piece_id=TO2LWDLQH7HBTNKDT4IQXIIUVRHE4LE4GGIUKPCN4A6QSU7RQFIQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.805306028 (830h43m49.676389063s) [noise=true , address=45.91.116.5:4300 , piece_id=2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.812815357 (830h43m49.683898392s) [noise=true , address=45.91.116.8:4047 , piece_id=IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.812835094 (830h43m49.683918129s) [seg_pos={0 0}, upload=726638]: Segment reached optimal threshold of 13 pieces. Jan 9 06:12:49.813052638 (830h43m49.684135683s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813060243 (830h43m49.684143278s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813246788 (830h43m49.684329823s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813262127 (830h43m49.684345162s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813290561 (830h43m49.684373596s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813297895 (830h43m49.68438093s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813318695 (830h43m49.68440173s) [seg_pos={0 0}, upload=726638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:12:49.813392696 (830h43m49.684475731s) [upload=726638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:12:49.813397244 (830h43m49.684480279s) [upload=726638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:12:50.098106208 (830h43m49.969189253s) [upload=726638]: Done uploading segments... Jan 9 06:12:50.098124663 (830h43m49.969207708s) [upload=726638]: Upload finished. err: <nil> Duration: 6.13622715s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 16.861117097s trace: https://jaeger.dev.storj.io/trace/2f53d28901977662 FAIL (2026-01-09 06:13:29 UTC) orbiter warning: context deadline exceeded Duration: 16.861141422s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXX.XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001330276s trace: https://jaeger.dev.storj.io/trace/1431e972fbb1d721 FAIL (2026-01-09 06:13:27 UTC) orbiter warning: context deadline exceeded Duration: 5.001352739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.X..................XX....X...........X........X.X...XX.X.] duration: 6.002306894s trace: https://jaeger.dev.storj.io/trace/592dc21013563f8f FAIL (2026-01-09 06:13:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q) context deadline exceeded ======LOGS====== Jan 9 06:12:47.163506303 (830h43m47.034589348s) [upload=726638]: Starting upload Jan 9 06:12:47.163564785 (830h43m47.03464782s) [upload=726638]: Done starting upload Jan 9 06:12:47.163741030 (830h43m47.034824065s) [upload=726638]: Uploading segments... Jan 9 06:12:47.163774524 (830h43m47.034857569s) [upload=726638]: Got next segment. Inline: false Jan 9 06:12:47.163799531 (830h43m47.034882576s) [upload=726638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:12:47.163804091 (830h43m47.034887126s) [upload=726638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:12:47.408937297 (830h43m47.280020332s) [seg_pos={0 0}, upload=726638]: Begin upload segment... Jan 9 06:12:47.408990427 (830h43m47.280073472s) [seg_pos={0 0}, upload=726638]: Done begin upload segment. Jan 9 06:12:47.409020505 (830h43m47.28010358s) [noise=true , address=145.79.227.23:4545 , piece_id=IZYOX3RMPU6GWWATG2LF4FGY6ORHOTV2SEYCK4CRNPIJIN5CDYCQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.409077524 (830h43m47.280160569s) [noise=true , address=145.79.227.3:4150 , piece_id=OOBMM5FL2JMFANGWZBRIXCIYBWJAJ3VP47PFCDVN3UTJHAYCYJBQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.410106573 (830h43m47.281189618s) [noise=true , address=45.91.116.5:4300 , piece_id=2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.410734779 (830h43m47.281817824s) [noise=true , address=145.79.227.42:4121 , piece_id=U52S7JVFLTKSEOA4JKHL4AGQ45SQZHW577WRRYGKXQO7H4DGKWZQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411197601 (830h43m47.282280646s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411688235 (830h43m47.28277128s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411917261 (830h43m47.283000296s) [noise=true , address=145.79.227.34:4905 , piece_id=I5O23IEXQYMJPVHNXH5YSGFP3ZAZZUZNWW7TOIZMWOASTPO6CJAQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412162217 (830h43m47.283245262s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412450186 (830h43m47.283533221s) [noise=true , address=45.91.116.8:4047 , piece_id=IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412592587 (830h43m47.283675632s) [noise=true , address=145.79.227.48:4303 , piece_id=OQBE6662QB447LECLADWKKFE573PF6OGJIJOOSSIE7CCSJNHWIMQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412764935 (830h43m47.28384798s) [noise=true , address=145.79.227.44:4193 , piece_id=HO3HEQQVVE5LBETZSEYQXG4GL5O33S6XDU4WDDHPYDXVSP7PDR7Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413020271 (830h43m47.284103316s) [noise=true , address=145.79.227.49:4337 , piece_id=I37AARED2Z2GGJ2FOG2K3425CCIEKN5EEFSPIYKTFUBPPZS73L6Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413124560 (830h43m47.284207605s) [noise=true , address=145.79.227.9:4367 , piece_id=QJXFC7CALWQ2IEGPKFLT6QGNYYPJ75BNKGANVICKG4VHV7ACUDPA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413464157 (830h43m47.284547202s) [noise=true , address=145.79.227.36:4519 , piece_id=F5SEZVZ3I2MW6XIM463IXSGKYCBIHHJE2XN4JNZ56BQZSL2VKW2Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.414009855 (830h43m47.28509289s) [noise=true , address=145.79.227.46:4601 , piece_id=TO2LWDLQH7HBTNKDT4IQXIIUVRHE4LE4GGIUKPCN4A6QSU7RQFIQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.414503566 (830h43m47.285586611s) [noise=true , address=145.79.227.38:4978 , piece_id=JGA3CCDJPAA6OKIT7IPQO25OQMK47OOCJDOMKP2TWN73WDE6F2AQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:49.005318296 (830h43m48.876401351s) [upload=726638]: Next returned nil segment Jan 9 06:12:49.005332112 (830h43m48.876415157s) [upload=726638]: Waiting for error group managing segments... Jan 9 06:12:49.386947948 (830h43m49.258030983s) [noise=true , address=145.79.227.36:4519 , piece_id=F5SEZVZ3I2MW6XIM463IXSGKYCBIHHJE2XN4JNZ56BQZSL2VKW2Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.397771488 (830h43m49.268854523s) [noise=true , address=145.79.227.3:4150 , piece_id=OOBMM5FL2JMFANGWZBRIXCIYBWJAJ3VP47PFCDVN3UTJHAYCYJBQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.399566335 (830h43m49.27064937s) [noise=true , address=145.79.227.49:4337 , piece_id=I37AARED2Z2GGJ2FOG2K3425CCIEKN5EEFSPIYKTFUBPPZS73L6Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.402655958 (830h43m49.273738993s) [noise=true , address=145.79.227.23:4545 , piece_id=IZYOX3RMPU6GWWATG2LF4FGY6ORHOTV2SEYCK4CRNPIJIN5CDYCQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.405514451 (830h43m49.276597496s) [noise=true , address=145.79.227.44:4193 , piece_id=HO3HEQQVVE5LBETZSEYQXG4GL5O33S6XDU4WDDHPYDXVSP7PDR7Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.415164376 (830h43m49.286247421s) [noise=true , address=145.79.227.34:4905 , piece_id=I5O23IEXQYMJPVHNXH5YSGFP3ZAZZUZNWW7TOIZMWOASTPO6CJAQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.416915090 (830h43m49.287998125s) [noise=true , address=145.79.227.48:4303 , piece_id=OQBE6662QB447LECLADWKKFE573PF6OGJIJOOSSIE7CCSJNHWIMQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.418002189 (830h43m49.289085224s) [noise=true , address=145.79.227.42:4121 , piece_id=U52S7JVFLTKSEOA4JKHL4AGQ45SQZHW577WRRYGKXQO7H4DGKWZQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.422796048 (830h43m49.293879083s) [noise=true , address=145.79.227.38:4978 , piece_id=JGA3CCDJPAA6OKIT7IPQO25OQMK47OOCJDOMKP2TWN73WDE6F2AQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.485495085 (830h43m49.35657812s) [noise=true , address=145.79.227.9:4367 , piece_id=QJXFC7CALWQ2IEGPKFLT6QGNYYPJ75BNKGANVICKG4VHV7ACUDPA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.486332329 (830h43m49.357415364s) [noise=true , address=145.79.227.46:4601 , piece_id=TO2LWDLQH7HBTNKDT4IQXIIUVRHE4LE4GGIUKPCN4A6QSU7RQFIQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.805306028 (830h43m49.676389063s) [noise=true , address=45.91.116.5:4300 , piece_id=2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.812815357 (830h43m49.683898392s) [noise=true , address=45.91.116.8:4047 , piece_id=IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.812835094 (830h43m49.683918129s) [seg_pos={0 0}, upload=726638]: Segment reached optimal threshold of 13 pieces. Jan 9 06:12:49.813052638 (830h43m49.684135683s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813060243 (830h43m49.684143278s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813246788 (830h43m49.684329823s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813262127 (830h43m49.684345162s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813290561 (830h43m49.684373596s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813297895 (830h43m49.68438093s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813318695 (830h43m49.68440173s) [seg_pos={0 0}, upload=726638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:12:49.813392696 (830h43m49.684475731s) [upload=726638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:12:49.813397244 (830h43m49.684480279s) [upload=726638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:12:50.098106208 (830h43m49.969189253s) [upload=726638]: Done uploading segments... Jan 9 06:12:50.098124663 (830h43m49.969207708s) [upload=726638]: Upload finished. err: <nil> Duration: 6.002321924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.X..................XX....X...........X........X.X...XX.X] duration: 30.004791882s trace: https://jaeger.dev.storj.io/trace/592dc21013563f8f FAIL (2026-01-09 06:13:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q) context deadline exceeded ======LOGS====== Jan 9 06:12:47.163506303 (830h43m47.034589348s) [upload=726638]: Starting upload Jan 9 06:12:47.163564785 (830h43m47.03464782s) [upload=726638]: Done starting upload Jan 9 06:12:47.163741030 (830h43m47.034824065s) [upload=726638]: Uploading segments... Jan 9 06:12:47.163774524 (830h43m47.034857569s) [upload=726638]: Got next segment. Inline: false Jan 9 06:12:47.163799531 (830h43m47.034882576s) [upload=726638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:12:47.163804091 (830h43m47.034887126s) [upload=726638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:12:47.408937297 (830h43m47.280020332s) [seg_pos={0 0}, upload=726638]: Begin upload segment... Jan 9 06:12:47.408990427 (830h43m47.280073472s) [seg_pos={0 0}, upload=726638]: Done begin upload segment. Jan 9 06:12:47.409020505 (830h43m47.28010358s) [noise=true , address=145.79.227.23:4545 , piece_id=IZYOX3RMPU6GWWATG2LF4FGY6ORHOTV2SEYCK4CRNPIJIN5CDYCQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.409077524 (830h43m47.280160569s) [noise=true , address=145.79.227.3:4150 , piece_id=OOBMM5FL2JMFANGWZBRIXCIYBWJAJ3VP47PFCDVN3UTJHAYCYJBQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.410106573 (830h43m47.281189618s) [noise=true , address=45.91.116.5:4300 , piece_id=2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.410734779 (830h43m47.281817824s) [noise=true , address=145.79.227.42:4121 , piece_id=U52S7JVFLTKSEOA4JKHL4AGQ45SQZHW577WRRYGKXQO7H4DGKWZQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411197601 (830h43m47.282280646s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411688235 (830h43m47.28277128s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.411917261 (830h43m47.283000296s) [noise=true , address=145.79.227.34:4905 , piece_id=I5O23IEXQYMJPVHNXH5YSGFP3ZAZZUZNWW7TOIZMWOASTPO6CJAQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412162217 (830h43m47.283245262s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412450186 (830h43m47.283533221s) [noise=true , address=45.91.116.8:4047 , piece_id=IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412592587 (830h43m47.283675632s) [noise=true , address=145.79.227.48:4303 , piece_id=OQBE6662QB447LECLADWKKFE573PF6OGJIJOOSSIE7CCSJNHWIMQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.412764935 (830h43m47.28384798s) [noise=true , address=145.79.227.44:4193 , piece_id=HO3HEQQVVE5LBETZSEYQXG4GL5O33S6XDU4WDDHPYDXVSP7PDR7Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413020271 (830h43m47.284103316s) [noise=true , address=145.79.227.49:4337 , piece_id=I37AARED2Z2GGJ2FOG2K3425CCIEKN5EEFSPIYKTFUBPPZS73L6Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413124560 (830h43m47.284207605s) [noise=true , address=145.79.227.9:4367 , piece_id=QJXFC7CALWQ2IEGPKFLT6QGNYYPJ75BNKGANVICKG4VHV7ACUDPA, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.413464157 (830h43m47.284547202s) [noise=true , address=145.79.227.36:4519 , piece_id=F5SEZVZ3I2MW6XIM463IXSGKYCBIHHJE2XN4JNZ56BQZSL2VKW2Q, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.414009855 (830h43m47.28509289s) [noise=true , address=145.79.227.46:4601 , piece_id=TO2LWDLQH7HBTNKDT4IQXIIUVRHE4LE4GGIUKPCN4A6QSU7RQFIQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:47.414503566 (830h43m47.285586611s) [noise=true , address=145.79.227.38:4978 , piece_id=JGA3CCDJPAA6OKIT7IPQO25OQMK47OOCJDOMKP2TWN73WDE6F2AQ, seg_pos={0 0}, upload=726638]: Uploading piece... Jan 9 06:12:49.005318296 (830h43m48.876401351s) [upload=726638]: Next returned nil segment Jan 9 06:12:49.005332112 (830h43m48.876415157s) [upload=726638]: Waiting for error group managing segments... Jan 9 06:12:49.386947948 (830h43m49.258030983s) [noise=true , address=145.79.227.36:4519 , piece_id=F5SEZVZ3I2MW6XIM463IXSGKYCBIHHJE2XN4JNZ56BQZSL2VKW2Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.397771488 (830h43m49.268854523s) [noise=true , address=145.79.227.3:4150 , piece_id=OOBMM5FL2JMFANGWZBRIXCIYBWJAJ3VP47PFCDVN3UTJHAYCYJBQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.399566335 (830h43m49.27064937s) [noise=true , address=145.79.227.49:4337 , piece_id=I37AARED2Z2GGJ2FOG2K3425CCIEKN5EEFSPIYKTFUBPPZS73L6Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.402655958 (830h43m49.273738993s) [noise=true , address=145.79.227.23:4545 , piece_id=IZYOX3RMPU6GWWATG2LF4FGY6ORHOTV2SEYCK4CRNPIJIN5CDYCQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.405514451 (830h43m49.276597496s) [noise=true , address=145.79.227.44:4193 , piece_id=HO3HEQQVVE5LBETZSEYQXG4GL5O33S6XDU4WDDHPYDXVSP7PDR7Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.415164376 (830h43m49.286247421s) [noise=true , address=145.79.227.34:4905 , piece_id=I5O23IEXQYMJPVHNXH5YSGFP3ZAZZUZNWW7TOIZMWOASTPO6CJAQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.416915090 (830h43m49.287998125s) [noise=true , address=145.79.227.48:4303 , piece_id=OQBE6662QB447LECLADWKKFE573PF6OGJIJOOSSIE7CCSJNHWIMQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.418002189 (830h43m49.289085224s) [noise=true , address=145.79.227.42:4121 , piece_id=U52S7JVFLTKSEOA4JKHL4AGQ45SQZHW577WRRYGKXQO7H4DGKWZQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.422796048 (830h43m49.293879083s) [noise=true , address=145.79.227.38:4978 , piece_id=JGA3CCDJPAA6OKIT7IPQO25OQMK47OOCJDOMKP2TWN73WDE6F2AQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.485495085 (830h43m49.35657812s) [noise=true , address=145.79.227.9:4367 , piece_id=QJXFC7CALWQ2IEGPKFLT6QGNYYPJ75BNKGANVICKG4VHV7ACUDPA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.486332329 (830h43m49.357415364s) [noise=true , address=145.79.227.46:4601 , piece_id=TO2LWDLQH7HBTNKDT4IQXIIUVRHE4LE4GGIUKPCN4A6QSU7RQFIQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.805306028 (830h43m49.676389063s) [noise=true , address=45.91.116.5:4300 , piece_id=2K5B6WGGDEUF42NPLRIGP7LRQMSXPSBNHYH7SJH4CHHVC4TTJJ6Q, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.812815357 (830h43m49.683898392s) [noise=true , address=45.91.116.8:4047 , piece_id=IDZUSSAPORDUHZAG5TTR3XBFD4LZOUEW47V4KDRWACYBKV6S4NNA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: <nil> Jan 9 06:12:49.812835094 (830h43m49.683918129s) [seg_pos={0 0}, upload=726638]: Segment reached optimal threshold of 13 pieces. Jan 9 06:12:49.813052638 (830h43m49.684135683s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813060243 (830h43m49.684143278s) [noise=true , address=45.91.116.10:4082 , piece_id=3CKF4T3OILKKYGVQFIUTGMSWNCSGGOHUO7YSFAIRFQE3XBAJFDOQ, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813246788 (830h43m49.684329823s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813262127 (830h43m49.684345162s) [noise=true , address=77.90.31.14:4664 , piece_id=BYEAATTILVLLN33TICPHCPVTATY4K7EPWK7NU7KTPWTN42JFVERA, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813290561 (830h43m49.684373596s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 9 06:12:49.813297895 (830h43m49.68438093s) [noise=true , address=45.91.116.14:4192 , piece_id=OWYX4SLGO4JDI6AZKCO2PV2AVHBXAZXFXDHDRIQGKDZTEHRNOHTQ, seg_pos={0 0}, upload=726638]: Long tail cancellation due to optimal threshold Jan 9 06:12:49.813318695 (830h43m49.68440173s) [seg_pos={0 0}, upload=726638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:12:49.813392696 (830h43m49.684475731s) [upload=726638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:12:49.813397244 (830h43m49.684480279s) [upload=726638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:12:50.098106208 (830h43m49.969189253s) [upload=726638]: Done uploading segments... Jan 9 06:12:50.098124663 (830h43m49.969207708s) [upload=726638]: Upload finished. err: <nil> Duration: 30.004813893s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001708666s trace: https://jaeger.dev.storj.io/trace/2965666d54cb354f FAIL (2026-01-09 06:13:04 UTC) orbiter warning: context deadline exceeded Duration: 5.001734303s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 19.039105202s trace: https://jaeger.dev.storj.io/trace/2f53d28901977662 FAIL (2026-01-09 06:12:55 UTC) orbiter warning: context deadline exceeded Duration: 19.039138455s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 17.541841136s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:12:49 UTC) orbiter warning: context deadline exceeded Duration: 17.541887426s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001929465s trace: https://jaeger.dev.storj.io/trace/7c5b9530939ac1fe FAIL (2026-01-09 06:12:10 UTC) orbiter warning: context deadline exceeded Duration: 5.00195245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.X..................XX....X...........X........X.X...XX.] duration: 6.362094752s trace: https://jaeger.dev.storj.io/trace/24963197aa6c935c FAIL (2026-01-09 06:11:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 7UB5UX3KVETLHZEH3XSG3BXFTDVFUKADSPB36SUDAM4SWZGZLLFA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: E3SF6VRFPOD5KVDSUEUV4L2CAG3WSNQRV2ZG5F6QB3HYALKAWXAQ) context deadline exceeded ======LOGS====== Jan 9 06:10:49.948489106 (830h41m49.819572141s) [upload=726608]: Starting upload Jan 9 06:10:49.948549661 (830h41m49.819632696s) [upload=726608]: Done starting upload Jan 9 06:10:49.948690850 (830h41m49.819773885s) [upload=726608]: Uploading segments... Jan 9 06:10:49.948793025 (830h41m49.81987606s) [upload=726608]: Got next segment. Inline: false Jan 9 06:10:49.948824926 (830h41m49.819907961s) [upload=726608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:10:49.948829955 (830h41m49.81991299s) [upload=726608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:10:50.192583964 (830h41m50.063666999s) [seg_pos={0 0}, upload=726608]: Begin upload segment... Jan 9 06:10:50.192652525 (830h41m50.06373557s) [seg_pos={0 0}, upload=726608]: Done begin upload segment. Jan 9 06:10:50.192686951 (830h41m50.063769986s) [noise=true , address=145.79.227.52:4446 , piece_id=PGRRRHAV5VDG4KQABNGCELQAR5R2RDAEF7U3G4HCGICEVWW7Y7KA, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.192782693 (830h41m50.063865738s) [noise=true , address=77.90.31.14:4661 , piece_id=FGKDWDZTHUZF7XALQVUDDGYIB32IXTTS77Z5PM6LJPCHFYTIHGNQ, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.193352829 (830h41m50.064435864s) [noise=true , address=145.79.227.42:4121 , piece_id=GU5EWOHK7OWQ5LXQWY4YONBHJ4KPDTREE45MYMTIUKIFQTRZHC6A, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.193418323 (830h41m50.064501368s) [noise=true , address=145.79.227.21:4474 , piece_id=JNWPMRP7GR5OGFP6BNETPW4ANBMINURJLGBJ3BT4YBUL75VM2KAA, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.193643002 (830h41m50.064726037s) [noise=true , address=145.79.227.36:4519 , piece_id=E3XIYRVV2ZRULBYBXIGS2SD5CZJMU2TSUNBSMIAO35IBXMXPARAA, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.193993699 (830h41m50.065076744s) [noise=true , address=145.79.227.14:4838 , piece_id=GXAUZIEKGV27SJN6BOPMGNF3EBIHG77X2ARU3B5TOYXHMCAGGRIQ, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.194053734 (830h41m50.065136769s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GBWIODPLAZ6EJPKFQUIAFPPCGOGD2QB7CRP645CI5GDJBVCY7A, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.194315953 (830h41m50.065398988s) [noise=true , address=145.79.227.39:4014 , piece_id=THCQUC2N5K27K42UNFU6HMFAUAHOOSMNTRV33YH5UOCA64OEFPDA, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.194516765 (830h41m50.06559981s) [noise=true , address=77.90.31.21:4877 , piece_id=7UB5UX3KVETLHZEH3XSG3BXFTDVFUKADSPB36SUDAM4SWZGZLLFA, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.194600534 (830h41m50.065683579s) [noise=true , address=145.79.227.49:4338 , piece_id=OWSKHZFBCPDO3M22EOYGH6KGZC2HL2656AO3KQ53B3EZJDM3EW5A, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.194954699 (830h41m50.066037734s) [noise=true , address=77.90.31.13:4629 , piece_id=P2UXUNFBCYK2QVZEPWVSGTMFTKWUUYUWYDKLEAIGCZ4LXJLXRZAA, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.194994956 (830h41m50.066078001s) [noise=true , address=45.91.116.5:4300 , piece_id=S6NBDXOBUUCYDNYSSUXAVVVWP5ZA7CZDX5BULHBQEJPWRKW6FFUQ, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.195502011 (830h41m50.066585076s) [noise=true , address=145.79.227.11:4732 , piece_id=7CI43BJXTWBL5LK5XOVPQKWUNJCT5AXS2UMRNY2XLB3DVGZQQVKQ, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.195569019 (830h41m50.066652054s) [noise=true , address=145.79.227.23:4547 , piece_id=IYBAC64ECX6ZYB35V6ADERVPJLIFUZHFI3N66YIZR4FBKZBMTCPA, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.196033843 (830h41m50.067116889s) [noise=true , address=145.79.227.51:4409 , piece_id=H7N2NFK2NAITFPRBUG3NITQ4OEU5RPOIQ4AZBJFLZVNVV73CVWUQ, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:50.196432152 (830h41m50.067515188s) [noise=true , address=77.90.31.16:4735 , piece_id=E3SF6VRFPOD5KVDSUEUV4L2CAG3WSNQRV2ZG5F6QB3HYALKAWXAQ, seg_pos={0 0}, upload=726608]: Uploading piece... Jan 9 06:10:51.896438099 (830h41m51.767521144s) [upload=726608]: Next returned nil segment Jan 9 06:10:51.896502812 (830h41m51.767585847s) [upload=726608]: Waiting for error group managing segments... Jan 9 06:10:52.178459795 (830h41m52.04954284s) [noise=true , address=145.79.227.11:4732 , piece_id=7CI43BJXTWBL5LK5XOVPQKWUNJCT5AXS2UMRNY2XLB3DVGZQQVKQ, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.183498720 (830h41m52.054581755s) [noise=true , address=145.79.227.51:4409 , piece_id=H7N2NFK2NAITFPRBUG3NITQ4OEU5RPOIQ4AZBJFLZVNVV73CVWUQ, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.184348238 (830h41m52.055431293s) [noise=true , address=145.79.227.21:4474 , piece_id=JNWPMRP7GR5OGFP6BNETPW4ANBMINURJLGBJ3BT4YBUL75VM2KAA, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.203617279 (830h41m52.074700314s) [noise=true , address=145.79.227.36:4519 , piece_id=E3XIYRVV2ZRULBYBXIGS2SD5CZJMU2TSUNBSMIAO35IBXMXPARAA, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.204366655 (830h41m52.07544969s) [noise=true , address=145.79.227.39:4014 , piece_id=THCQUC2N5K27K42UNFU6HMFAUAHOOSMNTRV33YH5UOCA64OEFPDA, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.204824086 (830h41m52.075907121s) [noise=true , address=145.79.227.42:4121 , piece_id=GU5EWOHK7OWQ5LXQWY4YONBHJ4KPDTREE45MYMTIUKIFQTRZHC6A, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.206346625 (830h41m52.077429671s) [noise=true , address=145.79.227.23:4547 , piece_id=IYBAC64ECX6ZYB35V6ADERVPJLIFUZHFI3N66YIZR4FBKZBMTCPA, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.211682115 (830h41m52.08276515s) [noise=true , address=145.79.227.52:4446 , piece_id=PGRRRHAV5VDG4KQABNGCELQAR5R2RDAEF7U3G4HCGICEVWW7Y7KA, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.269191541 (830h41m52.140274586s) [noise=true , address=145.79.227.14:4838 , piece_id=GXAUZIEKGV27SJN6BOPMGNF3EBIHG77X2ARU3B5TOYXHMCAGGRIQ, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.342246788 (830h41m52.213329823s) [noise=true , address=145.79.227.49:4338 , piece_id=OWSKHZFBCPDO3M22EOYGH6KGZC2HL2656AO3KQ53B3EZJDM3EW5A, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.472377673 (830h41m52.343460718s) [noise=true , address=77.90.31.13:4629 , piece_id=P2UXUNFBCYK2QVZEPWVSGTMFTKWUUYUWYDKLEAIGCZ4LXJLXRZAA, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.492278908 (830h41m52.363361943s) [noise=true , address=77.90.31.16:4735 , piece_id=E3SF6VRFPOD5KVDSUEUV4L2CAG3WSNQRV2ZG5F6QB3HYALKAWXAQ, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.588193251 (830h41m52.459276296s) [noise=true , address=77.90.31.21:4877 , piece_id=7UB5UX3KVETLHZEH3XSG3BXFTDVFUKADSPB36SUDAM4SWZGZLLFA, seg_pos={0 0}, upload=726608]: Done uploading piece. err: <nil> Jan 9 06:10:52.588218520 (830h41m52.459301555s) [seg_pos={0 0}, upload=726608]: Segment reached optimal threshold of 13 pieces. Jan 9 06:10:52.588471291 (830h41m52.459554336s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GBWIODPLAZ6EJPKFQUIAFPPCGOGD2QB7CRP645CI5GDJBVCY7A, seg_pos={0 0}, upload=726608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Jan 9 06:10:52.588480909 (830h41m52.459563944s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GBWIODPLAZ6EJPKFQUIAFPPCGOGD2QB7CRP645CI5GDJBVCY7A, seg_pos={0 0}, upload=726608]: Long tail cancellation due to optimal threshold Jan 9 06:10:52.588665501 (830h41m52.459748546s) [noise=true , address=45.91.116.5:4300 , piece_id=S6NBDXOBUUCYDNYSSUXAVVVWP5ZA7CZDX5BULHBQEJPWRKW6FFUQ, seg_pos={0 0}, upload=726608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 9 06:10:52.588681952 (830h41m52.459764987s) [noise=true , address=45.91.116.5:4300 , piece_id=S6NBDXOBUUCYDNYSSUXAVVVWP5ZA7CZDX5BULHBQEJPWRKW6FFUQ, seg_pos={0 0}, upload=726608]: Long tail cancellation due to optimal threshold Jan 9 06:10:52.588874649 (830h41m52.459957694s) [noise=true , address=77.90.31.14:4661 , piece_id=FGKDWDZTHUZF7XALQVUDDGYIB32IXTTS77Z5PM6LJPCHFYTIHGNQ, seg_pos={0 0}, upload=726608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:50766->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:50766->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50766->77.90.31.14:4661: use of closed network connection Jan 9 06:10:52.588884778 (830h41m52.459967813s) [noise=true , address=77.90.31.14:4661 , piece_id=FGKDWDZTHUZF7XALQVUDDGYIB32IXTTS77Z5PM6LJPCHFYTIHGNQ, seg_pos={0 0}, upload=726608]: Long tail cancellation due to optimal threshold Jan 9 06:10:52.588938741 (830h41m52.460021776s) [seg_pos={0 0}, upload=726608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:10:52.589035855 (830h41m52.4601189s) [upload=726608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:10:52.589043230 (830h41m52.460126275s) [upload=726608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:10:52.885691257 (830h41m52.756774292s) [upload=726608]: Done uploading segments... Jan 9 06:10:52.885709341 (830h41m52.756792377s) [upload=726608]: Upload finished. err: <nil> Duration: 6.362119819s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 17.629600022s trace: https://jaeger.dev.storj.io/trace/1d17c7c9f08684f5 FAIL (2026-01-09 06:11:52 UTC) orbiter warning: context deadline exceeded Duration: 17.629624558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000860549s trace: https://jaeger.dev.storj.io/trace/377b02cb75ad57d9 FAIL (2026-01-09 06:11:41 UTC) orbiter warning: context deadline exceeded Duration: 5.000885026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.441851705s trace: https://jaeger.dev.storj.io/trace/2f53d28901977662 FAIL (2026-01-09 06:11:32 UTC) orbiter warning: context deadline exceeded Duration: 7.441881761s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)