Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.954323265s FAIL (2025-07-27 04:10:21 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.954351641s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......................X...] duration: 5.734809307s FAIL (2025-07-27 04:09:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.73483665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.771994018s FAIL (2025-07-27 04:09:24 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.772020477s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.790775491s FAIL (2025-07-27 04:09:07 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.790808895s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......................X..] duration: 5.854109057s FAIL (2025-07-27 04:09:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854174433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.889874235s FAIL (2025-07-27 04:08:46 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.889907418s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......................X.] duration: 5.662942146s FAIL (2025-07-27 04:08:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.662968106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.880046994s FAIL (2025-07-27 04:07:59 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.880080537s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X......................X] duration: 30.23952028s FAIL (2025-07-27 04:07:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.239542181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.843113281s FAIL (2025-07-27 04:06:55 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.843130865s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.995557253s FAIL (2025-07-27 04:06:31 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.995608951s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.767301629s FAIL (2025-07-27 04:06:21 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.767322169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.737191801s FAIL (2025-07-27 04:06:04 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.737217361s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 17.612685439s FAIL (2025-07-27 04:05:38 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 17.612704434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.75876944s FAIL (2025-07-27 04:05:08 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.758817483s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.019715295s FAIL (2025-07-27 04:04:21 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 6.019737859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.642987328s FAIL (2025-07-27 04:04:08 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.643014481s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.742401244s FAIL (2025-07-27 04:03:55 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.742426694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.552245815s FAIL (2025-07-27 04:03:54 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 9.552271283s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.677390882s FAIL (2025-07-27 04:03:27 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.677418165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.865493436s FAIL (2025-07-27 04:03:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.865520068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.712740122s FAIL (2025-07-27 04:02:59 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.712770902s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 22.094857501s FAIL (2025-07-27 04:02:23 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 22.094884533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 3.762799945s FAIL (2025-07-27 04:02:11 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 3.76281847s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.894229359s FAIL (2025-07-27 04:01:39 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.894254578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.005592702s FAIL (2025-07-27 04:00:45 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 6.005639853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.694065023s FAIL (2025-07-27 03:59:52 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.694094199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.829406904s FAIL (2025-07-27 03:58:57 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.829434276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 21.487579666s FAIL (2025-07-27 03:58:30 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 21.487609835s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.83815448s FAIL (2025-07-27 03:57:24 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.838185459s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.047861284s FAIL (2025-07-27 03:56:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 8.047889889s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.71343278s FAIL (2025-07-27 03:55:48 UTC) orbiter warning: uplink: too many requests Duration: 5.713475271s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.674702177s FAIL (2025-07-27 03:55:24 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.674746722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.781937163s FAIL (2025-07-27 03:55:03 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.781969635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.859597186s FAIL (2025-07-27 03:54:45 UTC) orbiter warning: uplink: too many requests Duration: 5.859626764s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.808695226s FAIL (2025-07-27 03:54:15 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.808724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.641243792s FAIL (2025-07-27 03:54:13 UTC) orbiter warning: uplink: too many requests Duration: 5.641272307s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.692659003s FAIL (2025-07-27 03:53:35 UTC) orbiter warning: uplink: too many requests Duration: 5.692700843s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.816042744s FAIL (2025-07-27 03:53:31 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.81606661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.802641656s FAIL (2025-07-27 03:53:05 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.802666053s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.752615728s FAIL (2025-07-27 03:52:44 UTC) orbiter warning: uplink: too many requests Duration: 5.75264269s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.69548617s FAIL (2025-07-27 03:52:22 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.695507992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.709024276s FAIL (2025-07-27 03:52:04 UTC) orbiter warning: uplink: too many requests Duration: 5.709052098s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.809055476s FAIL (2025-07-27 03:51:22 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.80908315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.677431378s FAIL (2025-07-27 03:50:51 UTC) orbiter warning: uplink: too many requests Duration: 5.677459381s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.835970774s FAIL (2025-07-27 03:50:41 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 5.835999418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 29.040671044s FAIL (2025-07-27 03:50:18 UTC) orbiter warning: uplink: too many requests Duration: 29.040699857s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002346524s FAIL (2025-07-27 03:50:15 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3M4H7): protocol: read tcp 188.166.225.170:50972->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:49:05.155375441 (64h13m14.877938481s) [seg_pos={0 0}, upload=61579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:49:05.155460253 (64h13m14.878023283s) [upload=61579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:49:05.155464962 (64h13m14.878028002s) [upload=61579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:49:05.438891741 (64h13m15.161454862s) [upload=61579]: Done uploading segments... Jul 27 03:49:05.438911659 (64h13m15.161474699s) [upload=61579]: Upload finished. err: <nil> Jul 27 03:49:46.383392983 (64h13m56.105956023s) [upload=61591]: Starting upload Jul 27 03:49:46.383450764 (64h13m56.106013804s) [upload=61591]: Done starting upload Jul 27 03:49:46.383546076 (64h13m56.106109116s) [upload=61591]: Uploading segments... Jul 27 03:49:46.383692076 (64h13m56.106255116s) [upload=61591]: Got next segment. Inline: false Jul 27 03:49:46.383722945 (64h13m56.106285985s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 03:49:46.383728225 (64h13m56.106291265s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 03:49:46.614616334 (64h13m56.337179374s) [seg_pos={0 0}, upload=61591]: Begin upload segment... Jul 27 03:49:46.614660418 (64h13m56.337223458s) [seg_pos={0 0}, upload=61591]: Done begin upload segment. Jul 27 03:49:46.614680997 (64h13m56.337244037s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.614746382 (64h13m56.337309422s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615036708 (64h13m56.337599748s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615204188 (64h13m56.337767228s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615297747 (64h13m56.337860787s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615562083 (64h13m56.338125123s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615569888 (64h13m56.338132928s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615936740 (64h13m56.33849978s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.615972288 (64h13m56.338535328s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616200644 (64h13m56.338763674s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616364849 (64h13m56.338927889s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616556916 (64h13m56.339119956s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616730328 (64h13m56.339293368s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.616912226 (64h13m56.339475266s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617080007 (64h13m56.339643047s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:46.617952206 (64h13m56.340515246s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Uploading piece... Jul 27 03:49:48.365686017 (64h13m58.088249057s) [upload=61591]: Next returned nil segment Jul 27 03:49:48.365703291 (64h13m58.088266321s) [upload=61591]: Waiting for error group managing segments... Jul 27 03:49:48.646241203 (64h13m58.368804243s) [noise=true , address=77.90.31.15:4699 , piece_id=D3FMRFSCFJX55HGYGIFD3WOWFTEKB7LRZ7QMFYYYZCPJI7TUWGQQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.647765519 (64h13m58.370328559s) [noise=true , address=45.91.116.7:4009 , piece_id=KXB2M54BZUOGF4AWOILANLRH5XP7EIS5IT6VAPPMBLJ72JQ37G4Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.652753032 (64h13m58.375316072s) [noise=true , address=77.90.31.13:4628 , piece_id=VHOXVFMCVO3OLDDCO3QXNMZCPU5GM62TZ6XW4MKKVVHF2EFPHA3Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.683463059 (64h13m58.406026099s) [noise=true , address=45.91.116.10:4081 , piece_id=XMSLEB6II43C65K2INLYMEMX4GJYIF2GAK6NLJ5OBJPRAV7WIARQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.688306606 (64h13m58.410869646s) [noise=true , address=45.91.116.5:4297 , piece_id=CKHWVOQ2L4EDW2DMAFHP6HRUKY3B6SNHK3IY76P5LM6DMAVSEMXA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.714665789 (64h13m58.437228819s) [noise=true , address=77.90.31.21:4877 , piece_id=6TVZKIBCXKUK2XP3ZZOJ4KY5Q2QXBRHJYFJUAKE6M2NFPV5ORXRA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.719372735 (64h13m58.441935775s) [noise=true , address=77.90.31.18:4807 , piece_id=V3GF5J4VF7CAFJ6UFDYWO2JEXDE3FXPVO2XSATCTWUHI2VW6ZKUQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.720403868 (64h13m58.442966908s) [noise=true , address=45.91.116.11:4118 , piece_id=QNUQ7ODA5S2QIVIQRLTVXNOBKQ3PS2VI3EQBU2PW3RECKREV7PEA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.723207262 (64h13m58.445770302s) [noise=true , address=45.91.116.9:4261 , piece_id=XGVMLWNYV327RV6NBDDRNRT444NK3O3DE7TTQUMJINRFVICE4OLQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.738742175 (64h13m58.461305215s) [noise=true , address=45.91.116.7:4010 , piece_id=RBKLX6PHJ5UQJMLLFYJGJCYFGNU62TGKX5BYLB5DLYJV6AH6N54Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.787009004 (64h13m58.509572044s) [noise=true , address=45.91.116.8:4048 , piece_id=NIR5ASJETIGAQOVTCSOO2GL3XN57JQRZRMLDPI7TMHUP7N75KESQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:49:48.791698286 (64h13m58.514261326s) [noise=true , address=45.91.116.6:4975 , piece_id=CDEMMWUPSVEGTDURI22Z5IMHDIXAQ7BDJ4FVFC7A3KUHIDD2ZC6A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298558490 (64h14m24.02112153s) [noise=true , address=104.36.82.4:4801 , piece_id=U4CBVQBWNSQ3OAF35KOLED6BI44G2CYHXQ2SD2YBG3X7TMQM22IQ, seg_pos={0 0}, upload=61591]: Done uploading piece. err: <nil> Jul 27 03:50:14.298593106 (64h14m24.021156156s) [seg_pos={0 0}, upload=61591]: Segment reached optimal threshold of 13 pieces. Jul 27 03:50:14.298948115 (64h14m24.021511155s) [noise=true , address=104.36.82.12:4151 , piece_id=OUR7X4XVRKLXDZFAODGNOVX5WROGT7UVEWU4IC4V4GDOI2CB6A2Q, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:52738->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.298995366 (64h14m24.021558416s) [noise=true , address=104.36.82.6:4873 , piece_id=RLZXWI3OFD5HGN7S26M4OYNQB7I7WM2YBA3GG26SRPNZO3RNFA7A, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 188.166.225.170:42880->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299029180 (64h14m24.021592231s) [noise=true , address=104.36.82.2:4732 , piece_id=XROLUFCC4UGW3V7YL4DJYWYZ5LJBKRSZHKUK2KLFO3LPIHHGGOSA, seg_pos={0 0}, upload=61591]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:57178->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 03:50:14.299140213 (64h14m24.021703253s) [seg_pos={0 0}, upload=61591]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 03:50:14.299221669 (64h14m24.021784709s) [upload=61591]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 03:50:14.299226057 (64h14m24.021789097s) [upload=61591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 03:50:14.592452096 (64h14m24.315015136s) [upload=61591]: Done uploading segments... Jul 27 03:50:14.592467615 (64h14m24.315030645s) [upload=61591]: Upload finished. err: <nil> Duration: 30.002377363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 7.269995127s FAIL (2025-07-27 03:49:15 UTC) orbiter warning: uplink: too many requests Duration: 7.270020484s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.784726076s FAIL (2025-07-27 03:48:01 UTC) orbiter warning: uplink: too many requests Duration: 5.784752477s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)