Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..........X....................................] duration: 6.521258023s FAIL (2025-09-10 11:14:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.521289373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..........X...................................] duration: 6.034447225s FAIL (2025-09-10 11:14:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.034474365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..........X..................................] duration: 6.384212159s FAIL (2025-09-10 11:14:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.384254351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........X.................................] duration: 6.652112742s FAIL (2025-09-10 11:13:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.652143541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..........X................................] duration: 6.492135501s FAIL (2025-09-10 11:13:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.492163083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..........X...............................] duration: 6.284769061s FAIL (2025-09-10 11:12:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.284793287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..........X..............................] duration: 6.517910409s FAIL (2025-09-10 11:11:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.51793687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..........X.............................] duration: 6.610447728s FAIL (2025-09-10 11:10:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.610479589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..........X............................] duration: 6.05660658s FAIL (2025-09-10 11:10:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.056627633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..........X...........................] duration: 6.190681349s FAIL (2025-09-10 11:09:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.190706597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..........X..........................] duration: 6.349461174s FAIL (2025-09-10 11:08:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.349491332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..........X.........................] duration: 6.304140429s FAIL (2025-09-10 11:08:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.304175516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..........X........................] duration: 6.811910684s FAIL (2025-09-10 11:08:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.811936603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..........X.......................] duration: 7.169837149s FAIL (2025-09-10 11:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 7.169866495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..........X......................] duration: 6.680420278s FAIL (2025-09-10 11:07:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.680448863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........................X..........X.....................] duration: 6.209754982s FAIL (2025-09-10 11:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.209782394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX..........................X..........X....................] duration: 6.393745267s FAIL (2025-09-10 11:05:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.393766988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX..........................X..........X...................] duration: 6.723135948s FAIL (2025-09-10 11:04:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.723154614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX..........................X..........X..................] duration: 8.78830766s FAIL (2025-09-10 11:03:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 8.788335414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..XX..........................X..........X.................] duration: 6.153681117s FAIL (2025-09-10 11:03:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.153704943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..XX..........................X..........X................] duration: 6.138268198s FAIL (2025-09-10 11:02:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.138302654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..XX..........................X..........X...............] duration: 5.986153561s FAIL (2025-09-10 11:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 5.986182017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..XX..........................X..........X..............] duration: 6.814585451s FAIL (2025-09-10 11:01:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.81465809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..XX..........................X..........X.............] duration: 6.435115562s FAIL (2025-09-10 11:00:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.435147934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..XX..........................X..........X............] duration: 5.919676623s FAIL (2025-09-10 10:59:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 5.919701852s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..XX..........................X..........X...........] duration: 6.205626627s FAIL (2025-09-10 10:58:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.205678606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..XX..........................X..........X..........] duration: 6.376605402s FAIL (2025-09-10 10:57:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 6.376625592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......X..XX..........................X..........X.........] duration: 10.846097427s FAIL (2025-09-10 10:57:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 10.846121543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.......X..XX..........................X..........X........] duration: 19.422297648s FAIL (2025-09-10 10:57:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 19.422325991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.......X..XX..........................X..........X.......] duration: 19.059812136s FAIL (2025-09-10 10:56:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 19.05984033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.......X..XX..........................X..........X......] duration: 21.095737313s FAIL (2025-09-10 10:55:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 21.095763693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.......X..XX..........................X..........X.....] duration: 14.341629543s FAIL (2025-09-10 10:54:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 14.341651386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.......X..XX..........................X..........X....] duration: 15.908409938s FAIL (2025-09-10 10:53:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 15.908439754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....X.......X..XX..........................X..........X...] duration: 7.95860827s FAIL (2025-09-10 10:52:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 7.958635412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.....X.......X..XX..........................X..........X..] duration: 15.783013315s FAIL (2025-09-10 10:51:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 15.783030287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X.....X.......X..XX..........................X..........X.] duration: 24.125453257s FAIL (2025-09-10 10:51:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 24.125478815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.X.....X.......X..XX..........................X..........X] duration: 30.002744212s FAIL (2025-09-10 10:50:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:50:13.776391512 (93h21m49.815964642s) [upload=82204]: Starting upload Sep 10 10:50:13.776552861 (93h21m49.816126001s) [upload=82204]: Done starting upload Sep 10 10:50:13.776665286 (93h21m49.816238446s) [upload=82204]: Uploading segments... Sep 10 10:50:13.776782229 (93h21m49.816355349s) [upload=82204]: Got next segment. Inline: false Sep 10 10:50:13.776841233 (93h21m49.816414353s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:50:13.776847014 (93h21m49.816420134s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:50:14.036040506 (93h21m50.075613626s) [seg_pos={0 0}, upload=82204]: Begin upload segment... Sep 10 10:50:14.036088498 (93h21m50.075661618s) [seg_pos={0 0}, upload=82204]: Done begin upload segment. Sep 10 10:50:14.036111091 (93h21m50.075684211s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036160025 (93h21m50.075733145s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036462815 (93h21m50.076035935s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036601941 (93h21m50.076175061s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036721008 (93h21m50.076294128s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036886795 (93h21m50.076459915s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.036973051 (93h21m50.076546171s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037144539 (93h21m50.076717659s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037231906 (93h21m50.076805026s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037405799 (93h21m50.076978919s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037486383 (93h21m50.077059503s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037669373 (93h21m50.077242493s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037751270 (93h21m50.07732439s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.037936164 (93h21m50.077509284s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038122090 (93h21m50.07769521s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:14.038362589 (93h21m50.077935709s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Uploading piece... Sep 10 10:50:15.819692576 (93h21m51.859265706s) [upload=82204]: Next returned nil segment Sep 10 10:50:15.819714508 (93h21m51.859287638s) [upload=82204]: Waiting for error group managing segments... Sep 10 10:50:16.119069630 (93h21m52.15864275s) [noise=true , address=145.79.227.18:4005 , piece_id=LK4BO4WJAHZBVPUIPPRUS6GDNOVC2BLUFPLOY3LJQCUATSCRKLHA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.167840306 (93h21m52.207413426s) [noise=true , address=145.79.227.2:4115 , piece_id=5BBEHO4MTAHAWVRMILN2TRPMROWWBZFB7MWTSHSEI4IUHSS2ZP4Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.170953152 (93h21m52.210526272s) [noise=true , address=145.79.227.6:4258 , piece_id=VWGTG73WGGZYXYXMWH7U46QXKT65EAZMETFCJGLBGS4NA5JXMRVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.216760066 (93h21m52.256333186s) [noise=true , address=45.91.116.6:4973 , piece_id=KHYHJNNQMB27USXGA7MSB2LGN6JUTBHJOYKY3UNEJ55ZOXMMNT5Q, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.257860815 (93h21m52.297433935s) [noise=true , address=45.91.116.8:4048 , piece_id=CUFM562675QVRYW5ZKQ3KIW6VII4GPFBF7XZLWXTZA5FHRJSA5BA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.270351125 (93h21m52.309924246s) [noise=true , address=77.90.31.19:4841 , piece_id=3NCOKU2HO2GBXSGR2PDXA6WTAOOS6LT5LDPVKH4BJWAADCB6USIA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.274545001 (93h21m52.314118121s) [noise=true , address=77.90.31.15:4699 , piece_id=IZ74QX56Z7JGDF7PK4QVIJZOSYDBGRNT3H5HVRBHU3KXQBFTR7GQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.276306642 (93h21m52.315879762s) [noise=true , address=77.90.31.10:4518 , piece_id=PKM43ZRS6YSFYZE6FS7VBHMCPXNYWMOUQIVGF7RN2XRPNA2PIFXA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.301055911 (93h21m52.340629031s) [noise=true , address=45.91.116.4:4940 , piece_id=X7WPMGSAOYQYS7NVUWC6PM5B6QGAOOQFTBIEJYYC6W6CPHCWTN5A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.318308776 (93h21m52.357881896s) [noise=true , address=77.90.31.21:4878 , piece_id=FG5QVA2PJNBJJ653FLZWQDDJ624BNVNFT7GNOYDSBQLYWGIBKQVA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.320491983 (93h21m52.360065103s) [noise=true , address=77.90.31.21:4877 , piece_id=ZSGW5N3Q6LFDI2M5Y7WGK4BVWER4EANLQLNVUJ5VL56IWJTIQC7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:16.352008351 (93h21m52.391581471s) [noise=true , address=45.91.116.14:4191 , piece_id=FH4KFGYJAXILPKEPNTBIEKVKZQOJXWRASJGHJUEBT7EK3P45EF2A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181359912 (93h21m53.220933032s) [noise=true , address=77.90.31.16:4735 , piece_id=K2NNL7VGSYRLEF27UPQ4EIU35RHC5GXCBHU6H7DCPM2GVNBUAESQ, seg_pos={0 0}, upload=82204]: Done uploading piece. err: <nil> Sep 10 10:50:17.181387094 (93h21m53.220960204s) [seg_pos={0 0}, upload=82204]: Segment reached optimal threshold of 13 pieces. Sep 10 10:50:17.181651991 (93h21m53.221225111s) [noise=true , address=45.91.116.11:4117 , piece_id=TTT6RQELXKGHHPHAMTWN4XX2TU6HNHEQRXWUX24MQIXETZGBK64A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52190->45.91.116.11:4117: use of closed network connection Sep 10 10:50:17.181664686 (93h21m53.221237806s) [noise=true , address=45.91.116.7:4010 , piece_id=2FUN6MIACVQRYKMHZ6XRQO5T3VALDDDBVWOKB3YVCYIPZPIDHW7A, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181708680 (93h21m53.22128179s) [noise=true , address=45.91.116.10:4082 , piece_id=67ZQ73GX65TOBVSMMG4S72YTTJS6QXFXACVX4JDU234JQKY3AOWA, seg_pos={0 0}, upload=82204]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 10 10:50:17.181728438 (93h21m53.221301558s) [seg_pos={0 0}, upload=82204]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:50:17.181790135 (93h21m53.221363256s) [upload=82204]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:50:17.181843448 (93h21m53.221416568s) [upload=82204]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:50:17.532482896 (93h21m53.572056086s) [upload=82204]: Done uploading segments... Sep 10 10:50:17.532510939 (93h21m53.572084139s) [upload=82204]: Upload finished. err: <nil> Duration: 30.002767797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.X.....X.......X..XX..........................X..........] duration: 23.011626458s FAIL (2025-09-10 10:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 23.011652849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X.....X.......X..XX..........................X.........] duration: 28.210021852s FAIL (2025-09-10 10:49:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 28.210049354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X.....X.......X..XX..........................X........] duration: 20.600257642s FAIL (2025-09-10 10:48:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 20.600276859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.X.....X.......X..XX..........................X.......] duration: 17.41019518s FAIL (2025-09-10 10:47:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 17.410227203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.X.....X.......X..XX..........................X......] duration: 7.909530003s FAIL (2025-09-10 10:45:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 7.90955948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X.X.....X.......X..XX..........................X.....] duration: 15.632910167s FAIL (2025-09-10 10:45:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 15.632936316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X.X.....X.......X..XX..........................X....] duration: 20.77832092s FAIL (2025-09-10 10:44:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 20.778339115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X.X.....X.......X..XX..........................X...] duration: 20.977655918s FAIL (2025-09-10 10:43:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 20.97767276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X.X.....X.......X..XX..........................X..] duration: 19.562901222s FAIL (2025-09-10 10:43:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 19.562932353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X.X.....X.......X..XX..........................X.] duration: 27.468516526s FAIL (2025-09-10 10:42:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 27.468561873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X.X.....X.......X..XX..........................X] duration: 30.001530395s FAIL (2025-09-10 10:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 10 10:41:24.063189880 (93h13m0.102763s) [upload=82070]: Starting upload Sep 10 10:41:24.063249825 (93h13m0.102822945s) [upload=82070]: Done starting upload Sep 10 10:41:24.063311653 (93h13m0.102884793s) [upload=82070]: Uploading segments... Sep 10 10:41:24.063487218 (93h13m0.103060348s) [upload=82070]: Got next segment. Inline: false Sep 10 10:41:24.064149106 (93h13m0.103722226s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:41:24.064159325 (93h13m0.103732455s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:41:24.316824171 (93h13m0.356397341s) [seg_pos={0 0}, upload=82070]: Begin upload segment... Sep 10 10:41:24.316877333 (93h13m0.356450493s) [seg_pos={0 0}, upload=82070]: Done begin upload segment. Sep 10 10:41:24.316913502 (93h13m0.356486652s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317740124 (93h13m0.357313304s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.317875543 (93h13m0.357448664s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318375620 (93h13m0.35794874s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318387893 (93h13m0.357961013s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.318803800 (93h13m0.35837692s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319027558 (93h13m0.358600678s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319069528 (93h13m0.358642648s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319244122 (93h13m0.358817242s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319359503 (93h13m0.358932623s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319496244 (93h13m0.359069364s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319735753 (93h13m0.359308873s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.319986933 (93h13m0.359560053s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320212976 (93h13m0.359786096s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320435312 (93h13m0.360008432s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:24.320712061 (93h13m0.360285181s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Uploading piece... Sep 10 10:41:26.079070069 (93h13m2.118643189s) [upload=82070]: Next returned nil segment Sep 10 10:41:26.079089215 (93h13m2.118662335s) [upload=82070]: Waiting for error group managing segments... Sep 10 10:41:26.374861100 (93h13m2.41443422s) [noise=true , address=145.79.227.2:4115 , piece_id=UDLKLPYEOP2H7X26ORFEJDW2TFY3AAC6AE3PQ6ILECHGMXL6D5AA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.402682046 (93h13m2.442255216s) [noise=true , address=145.79.227.7:4294 , piece_id=I5YD5YZEEZINMBO7NLIPJTVK24NSNH33XAFOIB7LM2BLSAQCGDPQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.454026968 (93h13m2.493600088s) [noise=true , address=145.79.227.17:4969 , piece_id=BHRVSCZRH7KC7AABYOKJ6QLRADWENXA6DNBPQIGAXXGIOBSJ6H4Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.518964458 (93h13m2.558537578s) [noise=true , address=45.91.116.6:4973 , piece_id=HIQMLQYQQS3RJMKJP4CVGRXAFY5D4DLYR2F4A7VRTB65E6HKRYKQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.527284850 (93h13m2.56685797s) [noise=true , address=77.90.31.19:4843 , piece_id=FU7SNHFH2XVY6D6QS3SSLQON4AYBFQYBBN2ZGXGMQPTNMJ232G4A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.538335476 (93h13m2.577908596s) [noise=true , address=45.91.116.9:4263 , piece_id=R2LHP2GMW2BJ5ADTPLCMT5WDEB4QR3OI26YB2JDBQ2UDX3PW4VNA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.545932032 (93h13m2.585505153s) [noise=true , address=145.79.227.14:4837 , piece_id=MJEABDA3GNPAAKIPBLQ4JFPBI6KNTCN64TAC3FGB5MVPR37AXQEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.546247395 (93h13m2.585820515s) [noise=true , address=45.91.116.8:4047 , piece_id=SQQSYNN2EL6GYSQCRONJZKFVYYP23CNTSUKWKW6TFTR22RA5X6ZQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.566337359 (93h13m2.605910479s) [noise=true , address=45.91.116.14:4190 , piece_id=YX2F2X32B76CQJBGJQPLVXHJVXEE6B2YUHGL52XWRPMUC5AZLH2Q, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.571613836 (93h13m2.611187017s) [noise=true , address=77.90.31.10:4518 , piece_id=IERWE4SQVJT5VDABZ5HIFV7SR3W3XN4T6OR5SIR2UQAFV27TXN3A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.573282039 (93h13m2.612855159s) [noise=true , address=145.79.227.3:4151 , piece_id=AY4UPCNKJKGHBQPXWR2BAYKT6S3QQGUTV5VNMG64FSNKAL3V2KJA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.575723059 (93h13m2.615296179s) [noise=true , address=77.90.31.12:4592 , piece_id=U5AQRN7NGKW2NT67TZLH7L73TGUFF2IMQ3QO4TVTG2KSNDW6IFUQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663675160 (93h13m2.70324828s) [noise=true , address=77.90.31.17:4771 , piece_id=LPQSAJ4HNEBPVMJKKWFRQ2F7Q6PQCLW4HCM42OSKDHFDQ642RTEA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: <nil> Sep 10 10:41:26.663702853 (93h13m2.703275973s) [seg_pos={0 0}, upload=82070]: Segment reached optimal threshold of 13 pieces. Sep 10 10:41:26.663893809 (93h13m2.703466919s) [noise=true , address=77.90.31.14:4663 , piece_id=DRSMWYTWIJUTH5UZ7OAJFF7OTJINL7TSEHOPKAX434X2RGCIMK6A, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663918716 (93h13m2.703491836s) [noise=true , address=77.90.31.13:4627 , piece_id=5456DP36PC72MBZLMJWXTVOGMREO6LBMFN3TS5OWIAW4TLSYMPMA, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Sep 10 10:41:26.663979492 (93h13m2.703552612s) [noise=true , address=77.90.31.16:4734 , piece_id=AT5B7TAWDFIFWNA4QGMA7AY2ESLRK54A4UISHVK2S3VRT35AD3EQ, seg_pos={0 0}, upload=82070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37052->77.90.31.16:4734: use of closed network connection Sep 10 10:41:26.663999149 (93h13m2.703572269s) [seg_pos={0 0}, upload=82070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:41:26.664057321 (93h13m2.703630442s) [upload=82070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:41:26.664061058 (93h13m2.703634179s) [upload=82070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:41:26.966954129 (93h13m3.006527259s) [upload=82070]: Done uploading segments... Sep 10 10:41:26.966976512 (93h13m3.006549642s) [upload=82070]: Upload finished. err: <nil> Duration: 30.001546397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X.X.....X.......X..XX..........................] duration: 23.693907269s FAIL (2025-09-10 10:40:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 10 10:22:50.211574984 (92h54m26.251148104s) [upload=81787]: Starting upload Sep 10 10:22:50.211639216 (92h54m26.251212336s) [upload=81787]: Done starting upload Sep 10 10:22:50.211835202 (92h54m26.251408322s) [upload=81787]: Uploading segments... Sep 10 10:22:50.211863516 (92h54m26.251436636s) [upload=81787]: Got next segment. Inline: false Sep 10 10:22:50.211889686 (92h54m26.251462806s) [upload=81787]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:22:50.211893614 (92h54m26.251466734s) [upload=81787]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:22:50.506080177 (92h54m26.545653297s) [seg_pos={0 0}, upload=81787]: Begin upload segment... Sep 10 10:22:50.506120174 (92h54m26.545693294s) [seg_pos={0 0}, upload=81787]: Done begin upload segment. Sep 10 10:22:50.506151213 (92h54m26.545724333s) [noise=true , address=45.91.116.12:4225 , piece_id=VV5TBJCYZRS4J7E6AH6JPRIBP4NOQFJMUX7JAARAMY5PFQNHORYA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.506202902 (92h54m26.545776022s) [noise=true , address=77.90.31.16:4734 , piece_id=UALMD5YOYPIOKNAKCNDALTVT6FISPFKRRYGACR5CU7IGDE27PY3Q, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.506575315 (92h54m26.546148435s) [noise=true , address=77.90.31.14:4663 , piece_id=JYLXW52JXCGKHBE7YQPJNJJYW5L4UW3P6CK2L36AQAFOP2DD7JSQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.506714672 (92h54m26.546287792s) [noise=true , address=45.91.116.6:4973 , piece_id=BDQL4P64HUACDOK64FKB73D4GNXYRV2JTC6TR7SOTCZKBGE5PGGA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.506908572 (92h54m26.546481693s) [noise=true , address=45.91.116.5:4298 , piece_id=MN4Z6V4DXCR4LUQMHSY2RQAQ2YMWQW5FWRAKCY3KX4REYYXHBDGQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507177938 (92h54m26.546751058s) [noise=true , address=45.91.116.11:4119 , piece_id=JPR2O5PEJPK6QAN3KYY5CCLWL6PHP4ACPZBXEY232QVGNPVWDABA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507315621 (92h54m26.546888742s) [noise=true , address=45.91.116.14:4191 , piece_id=MEFJF75VRIMDMEBVUXPOZRMCA3ZVFG2TTVXG2FHUOR4V3Q73EACQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507440039 (92h54m26.547013159s) [noise=true , address=45.91.116.6:4974 , piece_id=VUNVXU3FOWFMZ2F7KFSVZOZ4SVWACQHJR77UX7SZU7TFRDT5W5OA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507676802 (92h54m26.547249922s) [noise=true , address=45.91.116.13:4155 , piece_id=RFYZT4QRYXZPPK4K6BW3PXHRHFQIDRE2MHZE2SLMDHIJNJW36VPA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507706860 (92h54m26.54727998s) [noise=true , address=45.91.116.7:4011 , piece_id=6Q3JGE6GJ3TPZG3FUJP76E7RHOBRGFR2PECWSRW3MZQBAXH7PD7Q, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507915650 (92h54m26.54748877s) [noise=true , address=77.90.31.12:4590 , piece_id=TOH2CZZLBXWXJMKU47KQEYMICHU7LQAUB5ZNGMU4XBIIAGB6FXUQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507936609 (92h54m26.547509729s) [noise=true , address=45.91.116.8:4048 , piece_id=OK33ULM2LGQHF7IWU52RSYU2HIPPGPZTCITMMH7L2K2JEVWRAMJQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.508131301 (92h54m26.547704422s) [noise=true , address=45.91.116.13:4154 , piece_id=LZFRAOUXL4UORVH7DDTQ7P2Q3RGTEUWURYE2GEVBHK4ES6URS5EQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.508153825 (92h54m26.547726945s) [noise=true , address=77.90.31.10:4517 , piece_id=PNXW6P46LBKFDWO3NJS3GCRZLGD3KCXMXRTP3QC4J4IO6CZA2SCQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.508343868 (92h54m26.547916988s) [noise=true , address=45.91.116.4:4938 , piece_id=NUZ3DMIA2GR5PHPKMHQZTMCKBYTTMDZ4POJO3SBLGRTYYN4S2KAA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.508408062 (92h54m26.547981182s) [noise=true , address=145.79.227.14:4838 , piece_id=S3FOMNZRRUHRXIOXPXDGRXUXRLKAU46BLL46LLZPADKKSW4WW2NA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:52.261608080 (92h54m28.3011812s) [upload=81787]: Next returned nil segment Sep 10 10:22:52.261625694 (92h54m28.301198824s) [upload=81787]: Waiting for error group managing segments... Sep 10 10:22:52.543015199 (92h54m28.582588319s) [noise=true , address=145.79.227.14:4838 , piece_id=S3FOMNZRRUHRXIOXPXDGRXUXRLKAU46BLL46LLZPADKKSW4WW2NA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.666667785 (92h54m28.706240905s) [noise=true , address=45.91.116.8:4048 , piece_id=OK33ULM2LGQHF7IWU52RSYU2HIPPGPZTCITMMH7L2K2JEVWRAMJQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.675061107 (92h54m28.714634227s) [noise=true , address=45.91.116.14:4191 , piece_id=MEFJF75VRIMDMEBVUXPOZRMCA3ZVFG2TTVXG2FHUOR4V3Q73EACQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.677410563 (92h54m28.716983683s) [noise=true , address=45.91.116.5:4298 , piece_id=MN4Z6V4DXCR4LUQMHSY2RQAQ2YMWQW5FWRAKCY3KX4REYYXHBDGQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.679518356 (92h54m28.719091476s) [noise=true , address=45.91.116.7:4011 , piece_id=6Q3JGE6GJ3TPZG3FUJP76E7RHOBRGFR2PECWSRW3MZQBAXH7PD7Q, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.686974423 (92h54m28.726547543s) [noise=true , address=45.91.116.11:4119 , piece_id=JPR2O5PEJPK6QAN3KYY5CCLWL6PHP4ACPZBXEY232QVGNPVWDABA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.687536209 (92h54m28.727109329s) [noise=true , address=77.90.31.14:4663 , piece_id=JYLXW52JXCGKHBE7YQPJNJJYW5L4UW3P6CK2L36AQAFOP2DD7JSQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.705472450 (92h54m28.74504557s) [noise=true , address=45.91.116.4:4938 , piece_id=NUZ3DMIA2GR5PHPKMHQZTMCKBYTTMDZ4POJO3SBLGRTYYN4S2KAA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.714207125 (92h54m28.753780245s) [noise=true , address=45.91.116.12:4225 , piece_id=VV5TBJCYZRS4J7E6AH6JPRIBP4NOQFJMUX7JAARAMY5PFQNHORYA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.725157579 (92h54m28.764730699s) [noise=true , address=77.90.31.12:4590 , piece_id=TOH2CZZLBXWXJMKU47KQEYMICHU7LQAUB5ZNGMU4XBIIAGB6FXUQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.736257289 (92h54m28.775830399s) [noise=true , address=45.91.116.13:4154 , piece_id=LZFRAOUXL4UORVH7DDTQ7P2Q3RGTEUWURYE2GEVBHK4ES6URS5EQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.758170191 (92h54m28.797743311s) [noise=true , address=45.91.116.6:4974 , piece_id=VUNVXU3FOWFMZ2F7KFSVZOZ4SVWACQHJR77UX7SZU7TFRDT5W5OA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.787448366 (92h54m28.827021486s) [noise=true , address=45.91.116.13:4155 , piece_id=RFYZT4QRYXZPPK4K6BW3PXHRHFQIDRE2MHZE2SLMDHIJNJW36VPA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.787477011 (92h54m28.827050131s) [seg_pos={0 0}, upload=81787]: Segment reached optimal threshold of 13 pieces. Sep 10 10:22:52.787632238 (92h54m28.827205358s) [noise=true , address=45.91.116.6:4973 , piece_id=BDQL4P64HUACDOK64FKB73D4GNXYRV2JTC6TR7SOTCZKBGE5PGGA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Sep 10 10:22:52.787854814 (92h54m28.827427924s) [noise=true , address=77.90.31.10:4517 , piece_id=PNXW6P46LBKFDWO3NJS3GCRZLGD3KCXMXRTP3QC4J4IO6CZA2SCQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Sep 10 10:22:52.787903547 (92h54m28.827476687s) [noise=true , address=77.90.31.16:4734 , piece_id=UALMD5YOYPIOKNAKCNDALTVT6FISPFKRRYGACR5CU7IGDE27PY3Q, seg_pos={0 0}, upload=81787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 10 10:22:52.787931110 (92h54m28.82750423s) [seg_pos={0 0}, upload=81787]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:22:52.788006515 (92h54m28.827579635s) [upload=81787]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:22:52.788012426 (92h54m28.827585546s) [upload=81787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:22:53.168594695 (92h54m29.208167815s) [upload=81787]: Done uploading segments... Sep 10 10:22:53.168612889 (92h54m29.208186009s) [upload=81787]: Upload finished. err: <nil> Duration: 23.693930565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X.X.....X.......X..XX.........................] duration: 24.142895505s FAIL (2025-09-10 10:40:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 10 10:22:50.211574984 (92h54m26.251148104s) [upload=81787]: Starting upload Sep 10 10:22:50.211639216 (92h54m26.251212336s) [upload=81787]: Done starting upload Sep 10 10:22:50.211835202 (92h54m26.251408322s) [upload=81787]: Uploading segments... Sep 10 10:22:50.211863516 (92h54m26.251436636s) [upload=81787]: Got next segment. Inline: false Sep 10 10:22:50.211889686 (92h54m26.251462806s) [upload=81787]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 10 10:22:50.211893614 (92h54m26.251466734s) [upload=81787]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 10 10:22:50.506080177 (92h54m26.545653297s) [seg_pos={0 0}, upload=81787]: Begin upload segment... Sep 10 10:22:50.506120174 (92h54m26.545693294s) [seg_pos={0 0}, upload=81787]: Done begin upload segment. Sep 10 10:22:50.506151213 (92h54m26.545724333s) [noise=true , address=45.91.116.12:4225 , piece_id=VV5TBJCYZRS4J7E6AH6JPRIBP4NOQFJMUX7JAARAMY5PFQNHORYA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.506202902 (92h54m26.545776022s) [noise=true , address=77.90.31.16:4734 , piece_id=UALMD5YOYPIOKNAKCNDALTVT6FISPFKRRYGACR5CU7IGDE27PY3Q, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.506575315 (92h54m26.546148435s) [noise=true , address=77.90.31.14:4663 , piece_id=JYLXW52JXCGKHBE7YQPJNJJYW5L4UW3P6CK2L36AQAFOP2DD7JSQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.506714672 (92h54m26.546287792s) [noise=true , address=45.91.116.6:4973 , piece_id=BDQL4P64HUACDOK64FKB73D4GNXYRV2JTC6TR7SOTCZKBGE5PGGA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.506908572 (92h54m26.546481693s) [noise=true , address=45.91.116.5:4298 , piece_id=MN4Z6V4DXCR4LUQMHSY2RQAQ2YMWQW5FWRAKCY3KX4REYYXHBDGQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507177938 (92h54m26.546751058s) [noise=true , address=45.91.116.11:4119 , piece_id=JPR2O5PEJPK6QAN3KYY5CCLWL6PHP4ACPZBXEY232QVGNPVWDABA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507315621 (92h54m26.546888742s) [noise=true , address=45.91.116.14:4191 , piece_id=MEFJF75VRIMDMEBVUXPOZRMCA3ZVFG2TTVXG2FHUOR4V3Q73EACQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507440039 (92h54m26.547013159s) [noise=true , address=45.91.116.6:4974 , piece_id=VUNVXU3FOWFMZ2F7KFSVZOZ4SVWACQHJR77UX7SZU7TFRDT5W5OA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507676802 (92h54m26.547249922s) [noise=true , address=45.91.116.13:4155 , piece_id=RFYZT4QRYXZPPK4K6BW3PXHRHFQIDRE2MHZE2SLMDHIJNJW36VPA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507706860 (92h54m26.54727998s) [noise=true , address=45.91.116.7:4011 , piece_id=6Q3JGE6GJ3TPZG3FUJP76E7RHOBRGFR2PECWSRW3MZQBAXH7PD7Q, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507915650 (92h54m26.54748877s) [noise=true , address=77.90.31.12:4590 , piece_id=TOH2CZZLBXWXJMKU47KQEYMICHU7LQAUB5ZNGMU4XBIIAGB6FXUQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.507936609 (92h54m26.547509729s) [noise=true , address=45.91.116.8:4048 , piece_id=OK33ULM2LGQHF7IWU52RSYU2HIPPGPZTCITMMH7L2K2JEVWRAMJQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.508131301 (92h54m26.547704422s) [noise=true , address=45.91.116.13:4154 , piece_id=LZFRAOUXL4UORVH7DDTQ7P2Q3RGTEUWURYE2GEVBHK4ES6URS5EQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.508153825 (92h54m26.547726945s) [noise=true , address=77.90.31.10:4517 , piece_id=PNXW6P46LBKFDWO3NJS3GCRZLGD3KCXMXRTP3QC4J4IO6CZA2SCQ, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.508343868 (92h54m26.547916988s) [noise=true , address=45.91.116.4:4938 , piece_id=NUZ3DMIA2GR5PHPKMHQZTMCKBYTTMDZ4POJO3SBLGRTYYN4S2KAA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:50.508408062 (92h54m26.547981182s) [noise=true , address=145.79.227.14:4838 , piece_id=S3FOMNZRRUHRXIOXPXDGRXUXRLKAU46BLL46LLZPADKKSW4WW2NA, seg_pos={0 0}, upload=81787]: Uploading piece... Sep 10 10:22:52.261608080 (92h54m28.3011812s) [upload=81787]: Next returned nil segment Sep 10 10:22:52.261625694 (92h54m28.301198824s) [upload=81787]: Waiting for error group managing segments... Sep 10 10:22:52.543015199 (92h54m28.582588319s) [noise=true , address=145.79.227.14:4838 , piece_id=S3FOMNZRRUHRXIOXPXDGRXUXRLKAU46BLL46LLZPADKKSW4WW2NA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.666667785 (92h54m28.706240905s) [noise=true , address=45.91.116.8:4048 , piece_id=OK33ULM2LGQHF7IWU52RSYU2HIPPGPZTCITMMH7L2K2JEVWRAMJQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.675061107 (92h54m28.714634227s) [noise=true , address=45.91.116.14:4191 , piece_id=MEFJF75VRIMDMEBVUXPOZRMCA3ZVFG2TTVXG2FHUOR4V3Q73EACQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.677410563 (92h54m28.716983683s) [noise=true , address=45.91.116.5:4298 , piece_id=MN4Z6V4DXCR4LUQMHSY2RQAQ2YMWQW5FWRAKCY3KX4REYYXHBDGQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.679518356 (92h54m28.719091476s) [noise=true , address=45.91.116.7:4011 , piece_id=6Q3JGE6GJ3TPZG3FUJP76E7RHOBRGFR2PECWSRW3MZQBAXH7PD7Q, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.686974423 (92h54m28.726547543s) [noise=true , address=45.91.116.11:4119 , piece_id=JPR2O5PEJPK6QAN3KYY5CCLWL6PHP4ACPZBXEY232QVGNPVWDABA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.687536209 (92h54m28.727109329s) [noise=true , address=77.90.31.14:4663 , piece_id=JYLXW52JXCGKHBE7YQPJNJJYW5L4UW3P6CK2L36AQAFOP2DD7JSQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.705472450 (92h54m28.74504557s) [noise=true , address=45.91.116.4:4938 , piece_id=NUZ3DMIA2GR5PHPKMHQZTMCKBYTTMDZ4POJO3SBLGRTYYN4S2KAA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.714207125 (92h54m28.753780245s) [noise=true , address=45.91.116.12:4225 , piece_id=VV5TBJCYZRS4J7E6AH6JPRIBP4NOQFJMUX7JAARAMY5PFQNHORYA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.725157579 (92h54m28.764730699s) [noise=true , address=77.90.31.12:4590 , piece_id=TOH2CZZLBXWXJMKU47KQEYMICHU7LQAUB5ZNGMU4XBIIAGB6FXUQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.736257289 (92h54m28.775830399s) [noise=true , address=45.91.116.13:4154 , piece_id=LZFRAOUXL4UORVH7DDTQ7P2Q3RGTEUWURYE2GEVBHK4ES6URS5EQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.758170191 (92h54m28.797743311s) [noise=true , address=45.91.116.6:4974 , piece_id=VUNVXU3FOWFMZ2F7KFSVZOZ4SVWACQHJR77UX7SZU7TFRDT5W5OA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.787448366 (92h54m28.827021486s) [noise=true , address=45.91.116.13:4155 , piece_id=RFYZT4QRYXZPPK4K6BW3PXHRHFQIDRE2MHZE2SLMDHIJNJW36VPA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: <nil> Sep 10 10:22:52.787477011 (92h54m28.827050131s) [seg_pos={0 0}, upload=81787]: Segment reached optimal threshold of 13 pieces. Sep 10 10:22:52.787632238 (92h54m28.827205358s) [noise=true , address=45.91.116.6:4973 , piece_id=BDQL4P64HUACDOK64FKB73D4GNXYRV2JTC6TR7SOTCZKBGE5PGGA, seg_pos={0 0}, upload=81787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Sep 10 10:22:52.787854814 (92h54m28.827427924s) [noise=true , address=77.90.31.10:4517 , piece_id=PNXW6P46LBKFDWO3NJS3GCRZLGD3KCXMXRTP3QC4J4IO6CZA2SCQ, seg_pos={0 0}, upload=81787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Sep 10 10:22:52.787903547 (92h54m28.827476687s) [noise=true , address=77.90.31.16:4734 , piece_id=UALMD5YOYPIOKNAKCNDALTVT6FISPFKRRYGACR5CU7IGDE27PY3Q, seg_pos={0 0}, upload=81787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 10 10:22:52.787931110 (92h54m28.82750423s) [seg_pos={0 0}, upload=81787]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 10 10:22:52.788006515 (92h54m28.827579635s) [upload=81787]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 10 10:22:52.788012426 (92h54m28.827585546s) [upload=81787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 10 10:22:53.168594695 (92h54m29.208167815s) [upload=81787]: Done uploading segments... Sep 10 10:22:53.168612889 (92h54m29.208186009s) [upload=81787]: Upload finished. err: <nil> Duration: 24.142911806s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)