Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........................X...X........................] duration: 16.304725308s FAIL (2025-07-27 04:40:56 UTC) orbiter warning: uplink: context deadline exceeded Duration: 16.3047582s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.779355708s FAIL (2025-07-27 04:40:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.779377419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [............................XX...X....X.....................] duration: 1.974733979s FAIL (2025-07-27 04:40:25 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.974757092s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.709683261s FAIL (2025-07-27 04:39:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.709713639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........................X...X.......................] duration: 5.829953488s FAIL (2025-07-27 04:39:31 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.829970803s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX...X....X....................] duration: 2.01852395s FAIL (2025-07-27 04:39:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.018553457s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.806424854s FAIL (2025-07-27 04:38:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.806453981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX...X....X...................] duration: 1.940114823s FAIL (2025-07-27 04:38:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.940139021s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........................X...X......................] duration: 5.696209802s FAIL (2025-07-27 04:38:32 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.696233598s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.667463145s FAIL (2025-07-27 04:38:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.667482221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX...X....X..................] duration: 2.019222537s FAIL (2025-07-27 04:37:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.019258276s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.786748111s FAIL (2025-07-27 04:37:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.786781526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........................X...X.....................] duration: 5.773850155s FAIL (2025-07-27 04:37:45 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.773932773s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [................................XX...X....X.................] duration: 2.08540205s FAIL (2025-07-27 04:37:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.085425856s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........................X...X....................] duration: 5.690099667s FAIL (2025-07-27 04:37:02 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.690130427s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.69684554s FAIL (2025-07-27 04:36:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.696866873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........................X...X...................] duration: 5.951632485s FAIL (2025-07-27 04:36:31 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.951656541s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX...X....X................] duration: 1.841224153s FAIL (2025-07-27 04:36:25 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.84124881s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........................X...X..................] duration: 5.818759651s FAIL (2025-07-27 04:36:02 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.818781892s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.796089616s FAIL (2025-07-27 04:35:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.796119715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX...X....X...............] duration: 1.833289053s FAIL (2025-07-27 04:35:36 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.833323148s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX...X....X..............] duration: 1.842613423s FAIL (2025-07-27 04:35:07 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.842648201s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 8.128749162s FAIL (2025-07-27 04:34:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.128816752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........................X...X.................] duration: 5.768866679s FAIL (2025-07-27 04:34:47 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.768896695s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..........................X...X................] duration: 5.866194507s FAIL (2025-07-27 04:34:20 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.866220727s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................................XX...X....X.............] duration: 1.833074336s FAIL (2025-07-27 04:33:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.833101698s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........................X...X...............] duration: 5.902159496s FAIL (2025-07-27 04:33:50 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.902200585s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.707820466s FAIL (2025-07-27 04:33:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.70784357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........................X...X..............] duration: 5.736939259s FAIL (2025-07-27 04:33:23 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.736972953s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......................X....................................] duration: 5.704688816s FAIL (2025-07-27 04:33:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.704716239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......................X...................................] duration: 5.749570022s FAIL (2025-07-27 04:33:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.749599248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XX...X....X............] duration: 1.841639143s FAIL (2025-07-27 04:32:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.84166422s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..........................X...X.............] duration: 5.792129734s FAIL (2025-07-27 04:32:53 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.792152909s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................................XX...X....X...........] duration: 1.826456794s FAIL (2025-07-27 04:32:17 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.826500688s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..........................X...X............] duration: 5.72688427s FAIL (2025-07-27 04:31:46 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.726913195s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XX...X....X..........] duration: 1.79980931s FAIL (2025-07-27 04:31:46 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.799837354s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X......................X..................................] duration: 5.747046057s FAIL (2025-07-27 04:31:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.747075413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................XX...X....X.........] duration: 1.83939523s FAIL (2025-07-27 04:31:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.839440517s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XX...X....X........] duration: 2.013949746s FAIL (2025-07-27 04:31:09 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.013975285s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X......................X.................................] duration: 5.752650697s FAIL (2025-07-27 04:30:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.752716282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..........................X...X...........] duration: 5.744979604s FAIL (2025-07-27 04:30:33 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.745006836s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......................X................................] duration: 5.712985358s FAIL (2025-07-27 04:30:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.713015834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XX...X....X.......] duration: 1.822476678s FAIL (2025-07-27 04:30:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.822500555s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X..........................X...X..........] duration: 5.81738757s FAIL (2025-07-27 04:30:05 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.817415866s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XX...X....X......] duration: 1.854160189s FAIL (2025-07-27 04:29:51 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.854185669s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......................X...............................] duration: 5.730385881s FAIL (2025-07-27 04:29:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.730414305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [............................................XX...X....X.....] duration: 2.066044206s FAIL (2025-07-27 04:29:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.066071609s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X..........................X...X.........] duration: 5.682185418s FAIL (2025-07-27 04:29:05 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.682209205s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......................X..............................] duration: 5.741343417s FAIL (2025-07-27 04:29:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.741364417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XX...X....X....] duration: 1.836228061s FAIL (2025-07-27 04:28:57 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.836263929s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)