Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................................................X..] duration: 5.03552278s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:29:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.035554402s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [..XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X.....] duration: 5.001416789s trace: https://jaeger.dev.storj.io/trace/1b9d4801244736ba FAIL (2025-12-09 14:29:53 UTC) orbiter warning: context deadline exceeded Duration: 5.001439893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [...XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X....] duration: 5.001865246s trace: https://jaeger.dev.storj.io/trace/1b9d4801244736ba FAIL (2025-12-09 14:29:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001909651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.838958531s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:29:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.838987426s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 10.712475627s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:29:31 UTC) orbiter warning: context deadline exceeded Duration: 10.712507467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.054870838s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:29:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.054899534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...................................................X.] duration: 5.149044247s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:29:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.149080376s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 43.952348471s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:29:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 43.952375062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.038562143s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:29:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.03859264s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.............X...X.X....X.....X........X..XX..XX...X....] duration: 8.509663522s trace: https://jaeger.dev.storj.io/trace/5653defd8f987fc1 FAIL (2025-12-09 14:28:57 UTC) orbiter warning: context deadline exceeded Duration: 8.509688991s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.716221036s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:28:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.716251122s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................................X] duration: 31.348175862s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:28:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 31.348200539s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 4.750316512s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:28:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.75034714s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [....XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X...] duration: 3.746381366s trace: https://jaeger.dev.storj.io/trace/1b9d4801244736ba FAIL (2025-12-09 14:28:26 UTC) orbiter warning: context deadline exceeded Duration: 3.746413878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..........X...X............X...............X...........] duration: 5.694634072s trace: https://jaeger.dev.storj.io/trace/ca262e51315117c FAIL (2025-12-09 14:28:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.694665262s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.938179605s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:28:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.938205805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 6.542426811s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:28:13 UTC) orbiter warning: context deadline exceeded Duration: 6.542453713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.731245574s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:28:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.731278086s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..........X...X............X...............X..........] duration: 5.785346254s trace: https://jaeger.dev.storj.io/trace/ca262e51315117c FAIL (2025-12-09 14:27:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.785387733s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.94245155s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:27:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.942481506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.726232191s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.726464753s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.............X...X.X....X.....X........X..XX..XX...X...] duration: 12.337957175s trace: https://jaeger.dev.storj.io/trace/5653defd8f987fc1 FAIL (2025-12-09 14:27:48 UTC) orbiter warning: context deadline exceeded Duration: 12.337984867s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.007149539s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:27:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.007171823s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.708104121s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:27:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.708135199s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [.....XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X..] duration: 4.949103241s trace: https://jaeger.dev.storj.io/trace/1b9d4801244736ba FAIL (2025-12-09 14:27:29 UTC) orbiter warning: context deadline exceeded Duration: 4.94917588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.948733918s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.948762953s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [X.....XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X.] duration: 3.654569736s trace: https://jaeger.dev.storj.io/trace/1b9d4801244736ba FAIL (2025-12-09 14:27:08 UTC) orbiter warning: context deadline exceeded Duration: 3.654622256s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 9.378418718s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:27:07 UTC) orbiter warning: context deadline exceeded Duration: 9.378478792s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.............X...X.X....X.....X........X..XX..XX...X..] duration: 26.214487162s trace: https://jaeger.dev.storj.io/trace/5653defd8f987fc1 FAIL (2025-12-09 14:26:58 UTC) orbiter warning: context deadline exceeded Duration: 26.21451208s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..........X...X............X...............X.........] duration: 5.962014868s trace: https://jaeger.dev.storj.io/trace/ca262e51315117c FAIL (2025-12-09 14:26:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.962044015s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XX.....XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X] duration: 5.000792529s trace: https://jaeger.dev.storj.io/trace/1b9d4801244736ba FAIL (2025-12-09 14:26:47 UTC) orbiter warning: context deadline exceeded Duration: 5.000826885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.07197093s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:26:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.072002601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.894529062s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:26:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.894555172s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 4.83443508s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:26:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.834466058s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..........X...X............X...............X........] duration: 5.827948759s trace: https://jaeger.dev.storj.io/trace/ca262e51315117c FAIL (2025-12-09 14:26:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.827978336s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXX.....XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....] duration: 4.911297021s trace: https://jaeger.dev.storj.io/trace/450678bf7518f633 FAIL (2025-12-09 14:26:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.911325434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 9.716807062s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:25:55 UTC) orbiter warning: context deadline exceeded Duration: 9.716833011s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X.............X...X.X....X.....X........X..XX..XX...X.] duration: 28.900043096s trace: https://jaeger.dev.storj.io/trace/5653defd8f987fc1 FAIL (2025-12-09 14:25:52 UTC) orbiter warning: context deadline exceeded Duration: 28.900062503s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration: 9.877065396s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:25:39 UTC) orbiter warning: context deadline exceeded Duration: 9.877165255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 4.820952576s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:25:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.820981292s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.174366653s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:25:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.17439684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X.............X...X.X....X.....X........X..XX..XX...X] duration: 30.001092493s trace: https://jaeger.dev.storj.io/trace/5653defd8f987fc1 FAIL (2025-12-09 14:25:23 UTC) orbiter warning: context deadline exceeded Duration: 30.00111738s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.735968304s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:25:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.735996497s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.XXX.....XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X....] duration: 4.56594981s trace: https://jaeger.dev.storj.io/trace/450678bf7518f633 FAIL (2025-12-09 14:25:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.565981468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..........X...X............X...............X.......] duration: 5.825768528s trace: https://jaeger.dev.storj.io/trace/ca262e51315117c FAIL (2025-12-09 14:24:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.825794317s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 9.25845151s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:24:59 UTC) orbiter warning: context deadline exceeded Duration: 9.258474513s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [..XXX.....XX.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X...] duration: 4.87390662s trace: https://jaeger.dev.storj.io/trace/450678bf7518f633 FAIL (2025-12-09 14:24:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.87392272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.119331113s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:24:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.119361853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.898427817s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:24:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.898470489s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..........X...X............X...............X......] duration: 5.95347089s trace: https://jaeger.dev.storj.io/trace/ca262e51315117c FAIL (2025-12-09 14:24:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.953542035s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)