Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..................XX...X....X...............................] duration: 2.155139407s FAIL (2025-07-27 04:48:05 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.155166989s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 5.701742901s FAIL (2025-07-27 04:47:42 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.701776805s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.680374536s FAIL (2025-07-27 04:47:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.680412579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 5.675479313s FAIL (2025-07-27 04:47:03 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.675523958s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...................XX...X....X..............................] duration: 1.863520072s FAIL (2025-07-27 04:46:52 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.863570588s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.964684178s FAIL (2025-07-27 04:46:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.964709769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 5.775270082s FAIL (2025-07-27 04:46:18 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.775294837s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................XX...X....X.............................] duration: 1.932061363s FAIL (2025-07-27 04:45:44 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.93208598s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.970056539s FAIL (2025-07-27 04:45:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.970088639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 5.781720243s FAIL (2025-07-27 04:45:17 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.781761072s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX...X....X............................] duration: 1.818759121s FAIL (2025-07-27 04:44:56 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.818787405s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 5.713778862s FAIL (2025-07-27 04:44:50 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.713806143s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................XX...X....X...........................] duration: 1.894621721s FAIL (2025-07-27 04:44:25 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.894651518s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.807524206s FAIL (2025-07-27 04:44:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.807548503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........................X...X............................] duration: 6.007107234s FAIL (2025-07-27 04:44:16 UTC) orbiter warning: uplink: context deadline exceeded Duration: 6.007125559s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................XX...X....X..........................] duration: 1.87002077s FAIL (2025-07-27 04:44:06 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.870061257s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.775408821s FAIL (2025-07-27 04:43:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.775437015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........................X...X...........................] duration: 5.686162462s FAIL (2025-07-27 04:43:33 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.686185948s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................XX...X....X.........................] duration: 1.822243738s FAIL (2025-07-27 04:43:18 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.822272893s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........................X...X..........................] duration: 5.683947866s FAIL (2025-07-27 04:43:02 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.683977802s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.775754919s FAIL (2025-07-27 04:43:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.775786138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX...X....X........................] duration: 1.836628529s FAIL (2025-07-27 04:42:55 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.836661662s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.729824039s FAIL (2025-07-27 04:42:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.729854378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........................X...X.........................] duration: 5.721921267s FAIL (2025-07-27 04:42:00 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.722008213s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........................XX...X....X.......................] duration: 2.074623446s FAIL (2025-07-27 04:41:50 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.074654064s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.707938628s FAIL (2025-07-27 04:41:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.707964759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........................XX...X....X......................] duration: 1.890177174s FAIL (2025-07-27 04:41:28 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.890206349s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.684337168s FAIL (2025-07-27 04:40:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.684367698s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........................X...X........................] duration: 16.304725308s FAIL (2025-07-27 04:40:56 UTC) orbiter warning: uplink: context deadline exceeded Duration: 16.3047582s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.779355708s FAIL (2025-07-27 04:40:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.779377419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [............................XX...X....X.....................] duration: 1.974733979s FAIL (2025-07-27 04:40:25 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.974757092s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.709683261s FAIL (2025-07-27 04:39:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.709713639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........................X...X.......................] duration: 5.829953488s FAIL (2025-07-27 04:39:31 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.829970803s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX...X....X....................] duration: 2.01852395s FAIL (2025-07-27 04:39:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.018553457s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.806424854s FAIL (2025-07-27 04:38:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.806453981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX...X....X...................] duration: 1.940114823s FAIL (2025-07-27 04:38:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.940139021s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........................X...X......................] duration: 5.696209802s FAIL (2025-07-27 04:38:32 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.696233598s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.667463145s FAIL (2025-07-27 04:38:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.667482221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX...X....X..................] duration: 2.019222537s FAIL (2025-07-27 04:37:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.019258276s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.786748111s FAIL (2025-07-27 04:37:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.786781526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........................X...X.....................] duration: 5.773850155s FAIL (2025-07-27 04:37:45 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.773932773s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [................................XX...X....X.................] duration: 2.08540205s FAIL (2025-07-27 04:37:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.085425856s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........................X...X....................] duration: 5.690099667s FAIL (2025-07-27 04:37:02 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.690130427s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.69684554s FAIL (2025-07-27 04:36:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.696866873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........................X...X...................] duration: 5.951632485s FAIL (2025-07-27 04:36:31 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.951656541s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX...X....X................] duration: 1.841224153s FAIL (2025-07-27 04:36:25 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.84124881s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........................X...X..................] duration: 5.818759651s FAIL (2025-07-27 04:36:02 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.818781892s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.796089616s FAIL (2025-07-27 04:35:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== eg_pos={0 0}, upload=61851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:44996->104.36.82.8:4969: use of closed network connection; conn closed: op Jul 27 04:06:18.948129439 (64h30m28.670692469s) [seg_pos={0 0}, upload=61851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:06:18.948205915 (64h30m28.670768955s) [upload=61851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:06:18.948210594 (64h30m28.670773634s) [upload=61851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:06:19.279379864 (64h30m29.001942904s) [upload=61851]: Done uploading segments... Jul 27 04:06:19.279399632 (64h30m29.001962672s) [upload=61851]: Upload finished. err: <nil> Jul 27 04:07:27.579155494 (64h31m37.301718524s) [upload=61870]: Starting upload Jul 27 04:07:27.579206722 (64h31m37.301769762s) [upload=61870]: Done starting upload Jul 27 04:07:27.579455168 (64h31m37.302018208s) [upload=61870]: Uploading segments... Jul 27 04:07:27.579491568 (64h31m37.302054608s) [upload=61870]: Got next segment. Inline: false Jul 27 04:07:27.579517648 (64h31m37.302080678s) [upload=61870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:07:27.579522056 (64h31m37.302085086s) [upload=61870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:07:27.825096204 (64h31m37.547659244s) [seg_pos={0 0}, upload=61870]: Begin upload segment... Jul 27 04:07:27.825141812 (64h31m37.547704852s) [seg_pos={0 0}, upload=61870]: Done begin upload segment. Jul 27 04:07:27.825162331 (64h31m37.547725361s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825201846 (64h31m37.547764886s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825534974 (64h31m37.548098014s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825679371 (64h31m37.548242411s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.825819929 (64h31m37.548382969s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826106387 (64h31m37.548669427s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826369791 (64h31m37.548932832s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826559304 (64h31m37.549122334s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.826813480 (64h31m37.54937651s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827061214 (64h31m37.549624254s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827301925 (64h31m37.549864965s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827531855 (64h31m37.550094895s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827744301 (64h31m37.550307331s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.827981745 (64h31m37.550544775s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828191607 (64h31m37.550754647s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:27.828390297 (64h31m37.550953337s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Uploading piece... Jul 27 04:07:29.454230378 (64h31m39.176793418s) [upload=61870]: Next returned nil segment Jul 27 04:07:29.454250215 (64h31m39.176813265s) [upload=61870]: Waiting for error group managing segments... Jul 27 04:07:29.840066160 (64h31m39.5626292s) [noise=true , address=45.91.116.13:4155 , piece_id=PEOIUMGY2TDSAHPUEODQQNXRAIPQNTAZQMCBXRDZZMBP3TP2W7HA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.841558656 (64h31m39.564121696s) [noise=true , address=45.91.116.7:4009 , piece_id=PIRLD5OLPNECAC7FJNNKBRNFOADTFZGS5PM26EJCDN2YJW3ZWCPQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.842190715 (64h31m39.564753745s) [noise=true , address=45.91.116.10:4081 , piece_id=HIK5FZ4QUPQRC2QABIJ3N3SZNRNED3HHGT5JLDT3N4OUXUU2SDJQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.845258725 (64h31m39.567821765s) [noise=true , address=45.91.116.11:4117 , piece_id=JOOIYQD7QFWNEEWN3P3EZVAXEFQUUZGMEW7P267C2Y3BYRCVS3WQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.858335094 (64h31m39.580898124s) [noise=true , address=45.91.116.8:4048 , piece_id=FNT354ZAYJHYZ3PZXMS6GVMKT5DZJJOZY7U6PRKFB7UFUFX2ZWFQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.916992263 (64h31m39.639555293s) [noise=true , address=77.90.31.14:4662 , piece_id=MPDAJCR5BR5S5T7DEUXUHZLPFOV52CVY4JEXGBTMJC5J3JOUFQNA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.925421321 (64h31m39.647984361s) [noise=true , address=45.91.116.9:4262 , piece_id=MYDINFMSFUQGTZT4WK5KXCMEYPRR6Z7DJJLRGDV3RXT4GLTLTNMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:29.964243449 (64h31m39.68680648s) [noise=true , address=77.90.31.16:4734 , piece_id=4NOOYMLNMITGQM7URAXZZRIYPFRGSRBNNTEY2JWS7DXMQED4STEQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.028494551 (64h31m39.751057591s) [noise=true , address=77.90.31.16:4735 , piece_id=AWKCFRD7IHWMGQI7WFVH5BGC7RB4PZQDD3TJ2SZQ4J4DMRJVTHXQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.034243541 (64h31m39.756806581s) [noise=true , address=77.90.31.18:4806 , piece_id=FK6M3K4TMNNH3SIIWNP5257AON2ULKYKYPNPA6PKBUYVKCLEICHA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.036682407 (64h31m39.759245447s) [noise=true , address=45.91.116.4:4940 , piece_id=DQ6PDAESHTJAWMNORQZPADRTV5R5LQYGXNVOOQFAKS77TZDNRA2Q, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:30.040210738 (64h31m39.762773778s) [noise=true , address=45.91.116.5:4298 , piece_id=4ZL6RMTQN3Z2IVDE4YWKH4RQDA3CA6NSY5SUM2CYFJQ33M3IZSWA, seg_pos={0 0}, upload=61870]: Done uploading piece. err: <nil> Jul 27 04:07:56.928044994 (64h32m6.650608034s) [noise=true , address=104.36.82.6:4875 , piece_id=7E772QGVJI5JI3H3O4MDRHYMXGMGFWRQYNJ7GKEKS7YYGJGFBEQQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:49034->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928157218 (64h32m6.650720258s) [noise=true , address=104.36.82.12:4150 , piece_id=YX7BWOWY4MZUHB4JZBTX2GSXN4IH6S6P7XMFYO2NO5IG6J7XZ5XQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:57300->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928219587 (64h32m6.650782617s) [noise=true , address=104.36.82.4:4802 , piece_id=STLAWLWO6X7VXNEUNMBVFJTLRRPZZ3RPWGV6R7UFV3IWSY235LMQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928238354 (64h32m6.650801414s) [noise=true , address=104.36.82.16:4295 , piece_id=JUZ7BA7JJBV2ZCLQZZNAE22KVK4MT34GJIYKM24GI5NO5LYMWDKQ, seg_pos={0 0}, upload=61870]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:60468->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:07:56.928281416 (64h32m6.650844457s) [seg_pos={0 0}, upload=61870]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:07:57.165863905 (64h32m6.888426945s) [upload=61870]: Done uploading segments... Jul 27 04:07:57.165891117 (64h32m6.888454157s) [upload=61870]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.796119715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX...X....X...............] duration: 1.833289053s FAIL (2025-07-27 04:35:36 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.833323148s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX...X....X..............] duration: 1.842613423s FAIL (2025-07-27 04:35:07 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.842648201s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)