Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X...........] duration: 5.713859172s FAIL (2025-07-26 22:19:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.713882777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X...........] duration: 5.831289588s FAIL (2025-07-26 22:19:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.831319845s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..X...........X.................] duration: 6.32574992s FAIL (2025-07-26 22:19:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.325773896s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.229018731s FAIL (2025-07-26 22:19:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== _id=K2JASMLMM64N56Z6GIBVWUH7QRRGMK7Q3O4LVKNQ5VN5NH4PZVKA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.678925147 (58h42m33.401488187s) [noise=true , address=79.127.219.38:4004 , piece_id=HGZBV6OMUZXQ5537YLVNSNNROCE72C6UGX6H3OVL7YXMOMNDQ36A, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.679267461 (58h42m33.401830501s) [noise=true , address=104.36.82.13:4204 , piece_id=ZTSC5GRH7BF5QC3D7SBNNVGCLCTHLIOXM2WYT2GWR5TDMXW3VSLQ, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.679544863 (58h42m33.402107903s) [noise=true , address=104.36.82.22:4528 , piece_id=I7ZPBCPV2JXI6S5XKXYGQDELORRHICC5G5JSDKAFRAMPL3VGOKIA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.679610518 (58h42m33.402173548s) [noise=true , address=104.36.82.16:4313 , piece_id=XOOU5SOA5DQJ6D3MOZKTPRFMR77BBIGTYGWLI2OMCXW6JJ4JE7VA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.679898519 (58h42m33.402461579s) [noise=true , address=79.127.201.215:4356 , piece_id=RHLFHZQJ2Z5RJSZIYXOIDILXFYVYDTFBNE46MPOW3VAGOPIQWX5Q, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680091658 (58h42m33.402654698s) [noise=true , address=79.127.205.245:4244 , piece_id=MALMEUEXLITFZLFR6GSO6IJNY5H7XIZMMHWOOG4RCFBHOXHGRG2Q, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680224172 (58h42m33.402787232s) [noise=true , address=79.127.205.234:4619 , piece_id=4XDWA465NYPBZG3CRAR3MPQ325AI6UMGN7NIVDQ35PNAP5S4NT7Q, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680541359 (58h42m33.403104409s) [noise=true , address=104.36.82.39:4748 , piece_id=H5HV5PPSWTNFERDGC4YTCVLIOL2HXSLHFLPOS4WT64TERYRTFL6A, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680567159 (58h42m33.403130219s) [noise=true , address=104.36.82.5:4869 , piece_id=Y4KYWSS67PHD4UASJFDQJV7GNFP6X3AOKQVVNOEZ23BIV5SY767Q, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680613346 (58h42m33.403176386s) [noise=true , address=64.49.13.138:4483 , piece_id=3T233IH7NYOMC5YOZ654WHJ3BFNQEXIVJQHLM5M6JD2PRRNCSGCA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680879546 (58h42m33.403442596s) [noise=true , address=79.127.205.246:4275 , piece_id=TGUMN64TQVEY62SRYG3WML65TPZO6OS7GTUER4JQ7LSUNDTXS3DA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680947156 (58h42m33.403510196s) [noise=true , address=104.36.82.53:4124 , piece_id=KJRXOP4RMHH5RAH6P3LWSFJVFZU6OYCEI3V5445FGJ3SN2UUX2KA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681113714 (58h42m33.403676755s) [noise=true , address=104.36.82.46:4488 , piece_id=TIHWYNS3GVJCKBOOK3JFJIT7L37255ASMHWTGFJCEKATL36FQ2RQ, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681348673 (58h42m33.403911714s) [noise=true , address=104.36.82.47:4536 , piece_id=PLAHWXGDAXPOLPELRK7GJZCSSGFGRJJPVKLYJ7T27QLW6Y73ZSDQ, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681380484 (58h42m33.403943524s) [noise=true , address=77.90.31.17:4774 , piece_id=FYW7VEOMUSGWF7I3IPSCPP7SOSN3VEBO7HCHKXU7HGLYYFUHA6KA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681411303 (58h42m33.403974333s) [noise=true , address=79.127.205.233:4647 , piece_id=7TDQDE5V63EFA4XBI4F2J4CIXU3NZHZN6PQSPIOIF2HTFYF6DRBQ, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681708111 (58h42m33.404271151s) [noise=true , address=77.90.31.16:4747 , piece_id=4BKBJM3EBPYF3CZGQBNGOMZIJQ42DPLYLE4GFXORWFG6XQIIBFUA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:24.768905801 (58h42m34.491468831s) [upload=56339]: Next returned nil segment Jul 26 22:18:24.768924407 (58h42m34.491487447s) [upload=56339]: Waiting for error group managing segments... Jul 26 22:18:25.078517169 (58h42m34.801080199s) [noise=true , address=79.127.205.245:4244 , piece_id=MALMEUEXLITFZLFR6GSO6IJNY5H7XIZMMHWOOG4RCFBHOXHGRG2Q, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.088054276 (58h42m34.810617316s) [noise=true , address=79.127.205.234:4619 , piece_id=4XDWA465NYPBZG3CRAR3MPQ325AI6UMGN7NIVDQ35PNAP5S4NT7Q, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.206373843 (58h42m34.928936873s) [noise=true , address=64.49.13.138:4483 , piece_id=3T233IH7NYOMC5YOZ654WHJ3BFNQEXIVJQHLM5M6JD2PRRNCSGCA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.420195831 (58h42m35.142758871s) [noise=true , address=79.127.205.246:4275 , piece_id=TGUMN64TQVEY62SRYG3WML65TPZO6OS7GTUER4JQ7LSUNDTXS3DA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.674385704 (58h42m35.396948744s) [noise=true , address=77.90.31.16:4747 , piece_id=4BKBJM3EBPYF3CZGQBNGOMZIJQ42DPLYLE4GFXORWFG6XQIIBFUA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.744652173 (58h42m35.467215213s) [noise=true , address=79.127.205.233:4647 , piece_id=7TDQDE5V63EFA4XBI4F2J4CIXU3NZHZN6PQSPIOIF2HTFYF6DRBQ, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:26.136907322 (58h42m35.859470352s) [noise=true , address=79.127.219.38:4004 , piece_id=HGZBV6OMUZXQ5537YLVNSNNROCE72C6UGX6H3OVL7YXMOMNDQ36A, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:37.804852967 (58h42m47.527416007s) [noise=true , address=79.127.201.215:4356 , piece_id=RHLFHZQJ2Z5RJSZIYXOIDILXFYVYDTFBNE46MPOW3VAGOPIQWX5Q, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:40.903611577 (58h42m50.626174607s) [noise=true , address=77.90.31.17:4774 , piece_id=FYW7VEOMUSGWF7I3IPSCPP7SOSN3VEBO7HCHKXU7HGLYYFUHA6KA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:19:22.591611584 (58h43m32.314174624s) [noise=true , address=104.36.82.22:4528 , piece_id=I7ZPBCPV2JXI6S5XKXYGQDELORRHICC5G5JSDKAFRAMPL3VGOKIA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1nDzCSeAbqSKpZUVGcu4NEj32jsoqojqSpzfpKYoXciXEjsTwo, address:104.36.82.22:4528): protocol: write tcp 188.166.225.170:48014->104.36.82.22:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:22.591784053 (58h43m32.314347104s) [noise=true , address=104.36.82.56:4256 , piece_id=K2JASMLMM64N56Z6GIBVWUH7QRRGMK7Q3O4LVKNQ5VN5NH4PZVKA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: write tcp 188.166.225.170:34668->104.36.82.56:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:22.591832116 (58h43m32.314395156s) [noise=true , address=104.36.82.13:4204 , piece_id=ZTSC5GRH7BF5QC3D7SBNNVGCLCTHLIOXM2WYT2GWR5TDMXW3VSLQ, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: write tcp 188.166.225.170:46316->104.36.82.13:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:22.591887001 (58h43m32.314450031s) [noise=true , address=104.36.82.46:4488 , piece_id=TIHWYNS3GVJCKBOOK3JFJIT7L37255ASMHWTGFJCEKATL36FQ2RQ, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: read tcp 188.166.225.170:45498->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:22.591989417 (58h43m32.314552457s) [noise=true , address=104.36.82.5:4869 , piece_id=Y4KYWSS67PHD4UASJFDQJV7GNFP6X3AOKQVVNOEZ23BIV5SY767Q, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:104.36.82.5:4869): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592065652 (58h43m32.314628692s) [noise=true , address=104.36.82.47:4536 , piece_id=PLAHWXGDAXPOLPELRK7GJZCSSGFGRJJPVKLYJ7T27QLW6Y73ZSDQ, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592191143 (58h43m32.314754183s) [noise=true , address=104.36.82.53:4124 , piece_id=KJRXOP4RMHH5RAH6P3LWSFJVFZU6OYCEI3V5445FGJ3SN2UUX2KA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592287257 (58h43m32.314850297s) [noise=true , address=104.36.82.39:4748 , piece_id=H5HV5PPSWTNFERDGC4YTCVLIOL2HXSLHFLPOS4WT64TERYRTFL6A, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1aZ4HiQxxV3qYJLYViMi7me9mRnPPBgPihW8fwzeMTKXnwbUb9, address:104.36.82.39:4748): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592567743 (58h43m32.315130783s) [noise=true , address=104.36.82.16:4313 , piece_id=XOOU5SOA5DQJ6D3MOZKTPRFMR77BBIGTYGWLI2OMCXW6JJ4JE7VA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:104.36.82.16:4313): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592587370 (58h43m32.31515042s) [seg_pos={0 0}, upload=56339]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:19:22.818960500 (58h43m32.54152354s) [upload=56339]: Done uploading segments... Jul 26 22:19:22.818986520 (58h43m32.54154956s) [upload=56339]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.229035542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X..........] duration: 6.047730745s FAIL (2025-07-26 22:19:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.047760101s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [....XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X..........] duration: 5.821891719s FAIL (2025-07-26 22:18:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.821978265s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X..X...........X................] duration: 5.976140637s FAIL (2025-07-26 22:18:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.976160817s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [...XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.........] duration: 6.504105325s FAIL (2025-07-26 22:18:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.504133508s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232130842s FAIL (2025-07-26 22:18:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 66EIXDB6SRXFZS62UN3DDFEG6VGE2L7QQMSSIDBEGDKRZYIF2SA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196026007 (58h41m32.918589047s) [noise=true , address=104.36.82.47:4523 , piece_id=6WEFNRREMKK2DDL3AJ6EOBDUL65NVRG5QPP67TPSNCQFG7Q7SGTQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196245748 (58h41m32.918808788s) [noise=true , address=104.36.82.46:4504 , piece_id=56AG7BBZYIRTPAZXBBXIXMNDMVDWIV7FZZLUUDEWPANYCVIPRWAQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196536875 (58h41m32.919099915s) [noise=true , address=79.127.219.48:4090 , piece_id=TU5ROLJUOEVUGQK7KRXQHUVZBOTRPPIV4ZRH4DAHPBJDET4JSMNA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196749150 (58h41m32.91931219s) [noise=true , address=79.127.205.245:4239 , piece_id=36ZMWZZSE7SUWSXLTGVPETZPF47R3IT4MG3CEREZYRUOCWUNHKJQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196975875 (58h41m32.919538925s) [noise=true , address=104.36.82.13:4199 , piece_id=AE7BEEVNGRXCBYQNDOAQWJDU4J6664F2RXUQFZAWBFZXU77ABWUQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.197158333 (58h41m32.919721373s) [noise=true , address=104.36.82.53:4126 , piece_id=RQWX4PKHX5O6CMPISMGBSAD2FCQMZUIE2NLAQH7PZJC3TOJM3TYA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.197382081 (58h41m32.919945111s) [noise=true , address=104.36.82.56:4230 , piece_id=JREE7B6OBC4MJQAGLC6Y72ADSC5BPKWBPUNDLMA56OJ4DGOSLQTQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.197635847 (58h41m32.920198887s) [noise=true , address=104.36.82.19:4403 , piece_id=6HHV56YAPJ4SYDI6FLYCT7QXSOB7LK3RW2QSRYCLU5AYBVCVXNLA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.197862281 (58h41m32.920425311s) [noise=true , address=104.36.82.52:4120 , piece_id=X6U4HJC4VNOQN5UDNHQQLWHUWTDVQ6CFXSOXH7Z2JNYYPM7XWZSA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.198047524 (58h41m32.920610564s) [noise=true , address=104.36.82.16:4321 , piece_id=XFRVYB3YKGZXMNQLD36HAOLKIPFZBJFKGFRGZ4XNMAH75QETHE3A, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:24.306114851 (58h41m34.028677891s) [upload=56324]: Next returned nil segment Jul 26 22:17:24.306135130 (58h41m34.02869817s) [upload=56324]: Waiting for error group managing segments... Jul 26 22:17:24.619782982 (58h41m34.342346022s) [noise=true , address=79.127.205.245:4239 , piece_id=36ZMWZZSE7SUWSXLTGVPETZPF47R3IT4MG3CEREZYRUOCWUNHKJQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:17:25.112253227 (58h41m34.834816267s) [noise=true , address=193.149.164.143:4696 , piece_id=SD276I3U4LO5UGN7ATRGEY3J6PEYKCQ53TMHXKGKJIEK3HW5BYLQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:17:28.157631716 (58h41m37.880194756s) [noise=true , address=156.146.43.225:4434 , piece_id=G66EIXDB6SRXFZS62UN3DDFEG6VGE2L7QQMSSIDBEGDKRZYIF2SA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:17:28.365760502 (58h41m38.088323542s) [noise=true , address=79.127.219.48:4090 , piece_id=TU5ROLJUOEVUGQK7KRXQHUVZBOTRPPIV4ZRH4DAHPBJDET4JSMNA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:17:29.925769153 (58h41m39.648332193s) [noise=true , address=79.127.201.215:4343 , piece_id=PIZKLF7GFJNT4XD4V5CXTVRRZX67LQ3MRVQCAGAT77EWAYEMDMGA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:18:22.104689925 (58h42m31.827252965s) [noise=true , address=104.36.82.53:4126 , piece_id=RQWX4PKHX5O6CMPISMGBSAD2FCQMZUIE2NLAQH7PZJC3TOJM3TYA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: write tcp 188.166.225.170:57708->104.36.82.53:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104743468 (58h42m31.827306508s) [noise=true , address=104.36.82.47:4523 , piece_id=6WEFNRREMKK2DDL3AJ6EOBDUL65NVRG5QPP67TPSNCQFG7Q7SGTQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:18:22.104759909 (58h42m31.827322939s) [noise=true , address=104.36.82.14:4255 , piece_id=HLIJHJKIWLZDD2P5SFZTTNATHCHKGDF6ROOV6KNEVGF5F7VDBZAA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:104.36.82.14:4255): protocol: write tcp 188.166.225.170:41178->104.36.82.14:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104801308 (58h42m31.827364348s) [noise=true , address=104.36.82.46:4504 , piece_id=56AG7BBZYIRTPAZXBBXIXMNDMVDWIV7FZZLUUDEWPANYCVIPRWAQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:38374->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104834241 (58h42m31.827397281s) [noise=true , address=104.36.82.56:4230 , piece_id=JREE7B6OBC4MJQAGLC6Y72ADSC5BPKWBPUNDLMA56OJ4DGOSLQTQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: write tcp 188.166.225.170:40952->104.36.82.56:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104868937 (58h42m31.827431977s) [noise=true , address=104.36.82.19:4403 , piece_id=6HHV56YAPJ4SYDI6FLYCT7QXSOB7LK3RW2QSRYCLU5AYBVCVXNLA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: write tcp 188.166.225.170:53990->104.36.82.19:4403: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104897903 (58h42m31.827460943s) [noise=true , address=104.36.82.9:4029 , piece_id=BGV32JJN2EJWGK5ENQU5ZVVO5QJLFNYC56PLE562KK5EJKZLC65A, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12snsP1y5fbxY8XrDoXihzQrsBxJ5Y38C2jSGhHtV1ukN426eiP, address:104.36.82.9:4029): protocol: write tcp 188.166.225.170:52076->104.36.82.9:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104932109 (58h42m31.827495149s) [noise=true , address=104.36.82.16:4321 , piece_id=XFRVYB3YKGZXMNQLD36HAOLKIPFZBJFKGFRGZ4XNMAH75QETHE3A, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae, address:104.36.82.16:4321): protocol: write tcp 188.166.225.170:49368->104.36.82.16:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104951756 (58h42m31.827514796s) [noise=true , address=104.36.82.13:4199 , piece_id=AE7BEEVNGRXCBYQNDOAQWJDU4J6664F2RXUQFZAWBFZXU77ABWUQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7, address:104.36.82.13:4199): protocol: write tcp 188.166.225.170:35932->104.36.82.13:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105003204 (58h42m31.827566244s) [noise=true , address=104.36.82.36:4618 , piece_id=FHDVYJTTGNWIOIIKTWW2W73NJGXUMKPNVB2XPXQVC5HQCW6N6MFQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:104.36.82.36:4618): protocol: write tcp 188.166.225.170:50378->104.36.82.36:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105049964 (58h42m31.827613004s) [noise=true , address=104.36.82.55:4199 , piece_id=NE36CQPKCGYLLUXVP34OWYM4D4JZ3NJYZ4VTZVJGUDCCT4J6BZEA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: write tcp 188.166.225.170:45378->104.36.82.55:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105055615 (58h42m31.827618655s) [noise=true , address=104.36.82.49:4006 , piece_id=ZTAGNKD74XZIMSVDVVCJH27NR5PAH3C56JGKOVMZZBFNHUW3W43A, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12oMAD2S6soQVtMpXxakmXz6DobyVZk9fQ3Lp4UxczMdQj5eHGW, address:104.36.82.49:4006): protocol: write tcp 188.166.225.170:53964->104.36.82.49:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105081294 (58h42m31.827644334s) [noise=true , address=104.36.82.52:4120 , piece_id=X6U4HJC4VNOQN5UDNHQQLWHUWTDVQ6CFXSOXH7Z2JNYYPM7XWZSA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 188.166.225.170:33994->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105118164 (58h42m31.827681204s) [seg_pos={0 0}, upload=56324]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:18:22.335351420 (58h42m32.05791446s) [upload=56324]: Done uploading segments... Jul 26 22:18:22.335368874 (58h42m32.057931914s) [upload=56324]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.232161891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.....XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.........] duration: 5.73263894s FAIL (2025-07-26 22:18:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.732670941s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X..X...........X...............] duration: 6.268740663s FAIL (2025-07-26 22:17:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.268760852s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [....XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X........] duration: 5.821600469s FAIL (2025-07-26 22:17:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.821624794s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X..X...........X..............] duration: 6.090158441s FAIL (2025-07-26 22:17:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.090179561s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X........] duration: 6.469878973s FAIL (2025-07-26 22:17:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.469901315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235372645s FAIL (2025-07-26 22:17:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== oise=true , address=64.49.13.138:4482 , piece_id=IIQUNJVK2ZKQM6E6SE62TJQP6AH2ALZWPTUFTOBQBIA46ITUH53Q, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726111329 (58h40m32.44867437s) [noise=true , address=104.36.82.56:4240 , piece_id=KPTRUPKVA2HICFT255KCGL7HPVXIJALAXAATHJ2U2LRRUN6MRI5Q, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726250676 (58h40m32.448813746s) [noise=true , address=104.36.82.9:4038 , piece_id=LOLIDLMYRWUFIAIWAFMGUFBINTZIYCQWDRMT7ZUM4PMIS27HFUKA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726381086 (58h40m32.448944126s) [noise=true , address=104.36.82.4:4815 , piece_id=HSYQ2TB7KQZBZBISESPDM5BDZZHZ336W52JCSJKVP5YRG2WFOVKA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726551522 (58h40m32.449114552s) [noise=true , address=104.36.82.47:4544 , piece_id=QJ5AKJAA6R2WUJZLCLVVDXTTXJKGXZIW7D72OMVWCC65MARZHZFA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726661161 (58h40m32.449224242s) [noise=true , address=193.149.164.143:4696 , piece_id=Z4SSXTYM3CG32CGQYEW5WK3RCZJMJ7MYYBCIB2GYSAJWZTYOBAYA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726835625 (58h40m32.449398665s) [noise=true , address=104.36.82.52:4110 , piece_id=7UGDQ6UOWC2CZREUXWOSSZW77IFFHIPVN6BIRYGXKQPOTF7BS2WA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726894247 (58h40m32.449457277s) [noise=true , address=104.36.82.14:4226 , piece_id=FZ4TYOEP7WTSBCIYCRTKENBZMWJOSY223TTEI5M4PCAA7HY7MEOQ, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.727079872 (58h40m32.449642912s) [noise=true , address=77.90.31.10:4525 , piece_id=JY2LGV4RKE26JRPBY5NDWUN7TQA2WZXG4QZOTJ2UVRTWO6JI26GQ, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.727098709 (58h40m32.449661749s) [noise=true , address=45.91.116.14:4222 , piece_id=7TXNNKDTNZYVWGJSRHYVPMOSLPUZZAJ62F5QX7LO73WR2NXGJFNQ, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.727300995 (58h40m32.449864035s) [noise=true , address=79.127.205.246:4283 , piece_id=MVGCJDBV7TRYOAJXXYG3MMHZFQLOUEW4VC67TGS7Q7IZY43BEFRA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.727516488 (58h40m32.450079518s) [noise=true , address=104.36.82.5:4864 , piece_id=ZPP26JUJFUBPEBL4QQ42WIT4PJNWPUFUTG3A4AXQUGRRZXNF3M5Q, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:23.832997277 (58h40m33.555560307s) [upload=56309]: Next returned nil segment Jul 26 22:16:23.833018188 (58h40m33.555581228s) [upload=56309]: Waiting for error group managing segments... Jul 26 22:16:24.148692978 (58h40m33.871256008s) [noise=true , address=79.127.205.246:4283 , piece_id=MVGCJDBV7TRYOAJXXYG3MMHZFQLOUEW4VC67TGS7Q7IZY43BEFRA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:24.244664336 (58h40m33.967227396s) [noise=true , address=64.49.13.138:4482 , piece_id=IIQUNJVK2ZKQM6E6SE62TJQP6AH2ALZWPTUFTOBQBIA46ITUH53Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:24.682786220 (58h40m34.40534926s) [noise=true , address=45.91.116.14:4222 , piece_id=7TXNNKDTNZYVWGJSRHYVPMOSLPUZZAJ62F5QX7LO73WR2NXGJFNQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:24.783583341 (58h40m34.506146381s) [noise=true , address=77.90.31.10:4525 , piece_id=JY2LGV4RKE26JRPBY5NDWUN7TQA2WZXG4QZOTJ2UVRTWO6JI26GQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:24.848011319 (58h40m34.570574359s) [noise=true , address=45.91.116.10:4100 , piece_id=A5IDMUPIVPRLS2OQE54WYOEMAF2BFDTGELGJJX4WNYX5AM4ZZJHQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:25.508948087 (58h40m35.231511127s) [noise=true , address=193.149.164.143:4696 , piece_id=Z4SSXTYM3CG32CGQYEW5WK3RCZJMJ7MYYBCIB2GYSAJWZTYOBAYA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:17:21.629292218 (58h41m31.351855248s) [noise=true , address=104.36.82.9:4038 , piece_id=LOLIDLMYRWUFIAIWAFMGUFBINTZIYCQWDRMT7ZUM4PMIS27HFUKA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12VTVM9YuevPA4vsWExjpyW5B9pqyEKKruUwnj1jNhE3j1ScCBg, address:104.36.82.9:4038): protocol: rpc: dial tcp 104.36.82.9:4038: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629336443 (58h41m31.351899483s) [noise=true , address=104.36.82.46:4501 , piece_id=ENPOEH4NO55CY7GKGWTKNZZM3IZEVYZ4HSITZCJOPWJE6S3NGS5Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 188.166.225.170:59472->104.36.82.46:4501: use of closed network connection; write tcp 188.166.225.170:59472->104.36.82.46:4501: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59472->104.36.82.46:4501: use of closed network connection Jul 26 22:17:21.629373313 (58h41m31.351936354s) [noise=true , address=104.36.82.52:4110 , piece_id=7UGDQ6UOWC2CZREUXWOSSZW77IFFHIPVN6BIRYGXKQPOTF7BS2WA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629390537 (58h41m31.351953567s) [noise=true , address=104.36.82.56:4240 , piece_id=KPTRUPKVA2HICFT255KCGL7HPVXIJALAXAATHJ2U2LRRUN6MRI5Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:129QqkFEzZdtzQ6HAKxPWxDNgDVUNJJLH59oRVXnHExrwkDECyf, address:104.36.82.56:4240): protocol: rpc: dial tcp 104.36.82.56:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629412589 (58h41m31.351975619s) [noise=true , address=104.36.82.14:4226 , piece_id=FZ4TYOEP7WTSBCIYCRTKENBZMWJOSY223TTEI5M4PCAA7HY7MEOQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:104.36.82.14:4226): protocol: rpc: dial tcp 104.36.82.14:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629426696 (58h41m31.351989736s) [noise=true , address=104.36.82.5:4864 , piece_id=ZPP26JUJFUBPEBL4QQ42WIT4PJNWPUFUTG3A4AXQUGRRZXNF3M5Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629457484 (58h41m31.352020525s) [noise=true , address=104.36.82.55:4200 , piece_id=KFAJFMI4S22TZAWTLZ3HLRSG2AO7D6XLHQR2ZYC6JM5OC3WPJIRQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: write tcp 188.166.225.170:57996->104.36.82.55:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629461783 (58h41m31.352024823s) [noise=true , address=104.36.82.50:4033 , piece_id=ZMWTGQALLTZU5CQO6SZEZBUXURM7IBXMFJQZOPQCEDWJSMEDV67Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1GhnCxqMdrDsaY3SrRs7VmHbSn8htrbQ3zJy2B5Nb77awMkCR6, address:104.36.82.50:4033): protocol: rpc: dial tcp 104.36.82.50:4033: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629498674 (58h41m31.352061704s) [noise=true , address=104.36.82.22:4533 , piece_id=ZZCVN53UFYRFPZ5Q6OZQUGLJE4S3PHXU4JEDYOA4LMQVCUJ4LYWQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12LXK5pbXavbPZh4YSuEkgk7wsRqwvcVEHv4w9FDRYqSkEU89VF, address:104.36.82.22:4533): protocol: rpc: dial tcp 104.36.82.22:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629505406 (58h41m31.352068446s) [noise=true , address=104.36.82.4:4815 , piece_id=HSYQ2TB7KQZBZBISESPDM5BDZZHZ336W52JCSJKVP5YRG2WFOVKA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:104.36.82.4:4815): protocol: rpc: dial tcp 104.36.82.4:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629512760 (58h41m31.3520758s) [noise=true , address=104.36.82.47:4544 , piece_id=QJ5AKJAA6R2WUJZLCLVVDXTTXJKGXZIW7D72OMVWCC65MARZHZFA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:104.36.82.47:4544): protocol: write tcp 188.166.225.170:40436->104.36.82.47:4544: use of closed network connection; write tcp 188.166.225.170:40436->104.36.82.47:4544: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40436->104.36.82.47:4544: use of closed network connection Jul 26 22:17:21.629536876 (58h41m31.352099917s) [noise=true , address=104.36.82.19:4432 , piece_id=L4AGQGOZC2CGULIEIGBCSFDVRYFOG7CJVHLUJOABHSRVD6AL4CBQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12oh9XZaiLhaZdMyq1cx88yrrG9C9sHtH3RTA6W2Ly39t23ArSQ, address:104.36.82.19:4432): protocol: rpc: dial tcp 104.36.82.19:4432: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629550764 (58h41m31.352113804s) [seg_pos={0 0}, upload=56309]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:17:21.862172550 (58h41m31.5847356s) [upload=56309]: Done uploading segments... Jul 26 22:17:21.862202046 (58h41m31.584765136s) [upload=56309]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.235398415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.079885847s FAIL (2025-07-26 22:17:10 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 2.079916325s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..X...........X.............] duration: 6.117555573s FAIL (2025-07-26 22:16:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.11757451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.999162382s FAIL (2025-07-26 22:16:50 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.999205874s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.......] duration: 25.537360956s FAIL (2025-07-26 22:16:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 25.537388087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.......] duration: 6.092028138s FAIL (2025-07-26 22:16:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.092058788s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235598673s FAIL (2025-07-26 22:16:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 19.51:4212 , piece_id=ZOCGFN2RIKHLSVJ3TP7XDYKVNFXUJDRBIGODBHGCE6KIMHWKJRGA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.632586689 (58h39m28.355149719s) [noise=true , address=104.36.82.56:4241 , piece_id=VPGRZQN5EN7U5AYKOSJ32WJV3T5DN7JLROH2JVPTSZGO7FAFKNKA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.632966185 (58h39m28.355529225s) [noise=true , address=104.36.82.52:4103 , piece_id=ORKV4CDOSGLMHJ6PUNDA57HQBXE5C4YB27IGWJOGVMFDJQILJSKA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.633074873 (58h39m28.355637903s) [noise=true , address=79.127.205.246:4284 , piece_id=TORDDF7JXLTMX2XTEZBK33FAEZNWTO2UAXTASAG5YLGJ2UV46TFA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.633481221 (58h39m28.356044261s) [noise=true , address=104.36.82.45:4937 , piece_id=7CNFHQG7I56N42H5IPA2UXOP3CRFM6E4HGYDDFSZQNEIGHDR4H4A, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.633666655 (58h39m28.356229695s) [noise=true , address=104.36.82.47:4539 , piece_id=4B5CTYR56TG63BWAGRI7ONMEQVJPFBWICVWI4QE54I6SIJUAUTUA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.634074675 (58h39m28.356637715s) [noise=true , address=79.127.219.48:4090 , piece_id=CZJJV6VFNQKFBY6O6BZF5IVN6EZJJTIZRWD6PB6YJ44X4LLJRZAA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.634096898 (58h39m28.356659938s) [noise=true , address=79.127.219.38:4007 , piece_id=G7UG364ZLUWWJGRSD2KB34XR57JRPYPPUI5YN57HQDLZTWUEXFEQ, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.634508395 (58h39m28.357071436s) [noise=true , address=104.36.82.18:4370 , piece_id=AZUNHNW5C3F7KZTHXZWKG63XPSMSRWBHO6RWUMFHV7TT6EWTJEDA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.634614759 (58h39m28.357177789s) [noise=true , address=104.36.82.9:4022 , piece_id=LOQ743ADRSZKAZZTG566EZUCFXY4XN6ZOLR5FIDHRQD6ICCNYVKA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.635239104 (58h39m28.357802144s) [noise=true , address=77.90.31.13:4644 , piece_id=KSZEKKW6LKFV5BXAY2EPMU7WQ2K5IEYAJKUFNA4OVSWCF75XFC5Q, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.635460307 (58h39m28.358023347s) [noise=true , address=156.146.43.225:4434 , piece_id=APB2HOBJMCZJBLOPJKL2SGEVC5X6YA2ZGI3SOFKZDC32T6XYRRLQ, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.635657063 (58h39m28.358220104s) [noise=true , address=104.36.82.6:4903 , piece_id=B7RI6UHZCTBQRML42OXXL6FGPWD6CDVM2PJH2YTHVK6HWQ5SWNXQ, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:19.785734719 (58h39m29.508297759s) [upload=56288]: Next returned nil segment Jul 26 22:15:19.785751402 (58h39m29.508314442s) [upload=56288]: Waiting for error group managing segments... Jul 26 22:15:20.077111762 (58h39m29.799674802s) [noise=true , address=79.127.205.246:4284 , piece_id=TORDDF7JXLTMX2XTEZBK33FAEZNWTO2UAXTASAG5YLGJ2UV46TFA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:20.632258635 (58h39m30.354821675s) [noise=true , address=77.90.31.13:4644 , piece_id=KSZEKKW6LKFV5BXAY2EPMU7WQ2K5IEYAJKUFNA4OVSWCF75XFC5Q, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:20.877974995 (58h39m30.600538035s) [noise=true , address=79.127.219.38:4007 , piece_id=G7UG364ZLUWWJGRSD2KB34XR57JRPYPPUI5YN57HQDLZTWUEXFEQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:21.185022439 (58h39m30.907585479s) [noise=true , address=79.127.219.48:4090 , piece_id=CZJJV6VFNQKFBY6O6BZF5IVN6EZJJTIZRWD6PB6YJ44X4LLJRZAA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:24.671453279 (58h39m34.394016319s) [noise=true , address=79.127.219.51:4212 , piece_id=ZOCGFN2RIKHLSVJ3TP7XDYKVNFXUJDRBIGODBHGCE6KIMHWKJRGA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:25.874699680 (58h39m35.59726272s) [noise=true , address=156.146.43.225:4434 , piece_id=APB2HOBJMCZJBLOPJKL2SGEVC5X6YA2ZGI3SOFKZDC32T6XYRRLQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:26.710314392 (58h39m36.432877432s) [noise=true , address=79.127.201.215:4351 , piece_id=EJC6HH266U4KVMIXOG5DJHWPSDL4BEIAAPGNPJADLYY3HI4RB7UQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:16:17.545961274 (58h40m27.268524314s) [noise=true , address=104.36.82.9:4022 , piece_id=LOQ743ADRSZKAZZTG566EZUCFXY4XN6ZOLR5FIDHRQD6ICCNYVKA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: write tcp 188.166.225.170:54868->104.36.82.9:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546011050 (58h40m27.26857409s) [noise=true , address=104.36.82.53:4128 , piece_id=IVQWI4UQ7UYP3NBP4AFCTA7VLLJCUJX2ZYQVYOSXZFQZEWFKEKVQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 188.166.225.170:38354->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546034475 (58h40m27.268597515s) [noise=true , address=104.36.82.34:4552 , piece_id=TIZKHAFVGSKWE422F66JI6KBEYPJZMUEDWRZ7C3JTO2ZJBAPUWHA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:12YmWSZZQgXm2gdZCahnyCdSzxYPAerzCfD38mWPmpTDyGYT823, address:104.36.82.34:4552): protocol: write tcp 188.166.225.170:58424->104.36.82.34:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546100711 (58h40m27.268663751s) [noise=true , address=104.36.82.46:4483 , piece_id=HANC4VCEXEXBKNTYX7FJF6N3KT2FOREABTK72AV2THJZUS2WWHFQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: write tcp 188.166.225.170:52922->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546150797 (58h40m27.268713837s) [noise=true , address=104.36.82.47:4539 , piece_id=4B5CTYR56TG63BWAGRI7ONMEQVJPFBWICVWI4QE54I6SIJUAUTUA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: write tcp 188.166.225.170:48568->104.36.82.47:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546184010 (58h40m27.26874705s) [noise=true , address=104.36.82.56:4241 , piece_id=VPGRZQN5EN7U5AYKOSJ32WJV3T5DN7JLROH2JVPTSZGO7FAFKNKA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: write tcp 188.166.225.170:45984->104.36.82.56:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546317967 (58h40m27.268880997s) [noise=true , address=104.36.82.45:4937 , piece_id=7CNFHQG7I56N42H5IPA2UXOP3CRFM6E4HGYDDFSZQNEIGHDR4H4A, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94, address:104.36.82.45:4937): protocol: write tcp 188.166.225.170:47492->104.36.82.45:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546354637 (58h40m27.268917677s) [noise=true , address=104.36.82.18:4370 , piece_id=AZUNHNW5C3F7KZTHXZWKG63XPSMSRWBHO6RWUMFHV7TT6EWTJEDA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm, address:104.36.82.18:4370): protocol: write tcp 188.166.225.170:34922->104.36.82.18:4370: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546372291 (58h40m27.268935331s) [noise=true , address=104.36.82.52:4103 , piece_id=ORKV4CDOSGLMHJ6PUNDA57HQBXE5C4YB27IGWJOGVMFDJQILJSKA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 188.166.225.170:53508->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546394033 (58h40m27.268957073s) [noise=true , address=104.36.82.55:4212 , piece_id=GJ7T2EMA7IECPRT4GUPGB6U43T73MBIJEMP4XZGFF4VMYWECB6VA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: write tcp 188.166.225.170:60236->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546399072 (58h40m27.268962112s) [noise=true , address=104.36.82.6:4903 , piece_id=B7RI6UHZCTBQRML42OXXL6FGPWD6CDVM2PJH2YTHVK6HWQ5SWNXQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1zFBuDaxAtbFho2wumoXWSoQbB7y9VaSFizz31gV8n1fLK6syE, address:104.36.82.6:4903): protocol: write tcp 188.166.225.170:52932->104.36.82.6:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546418780 (58h40m27.26898182s) [seg_pos={0 0}, upload=56288]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:16:17.779435262 (58h40m27.501998302s) [upload=56288]: Done uploading segments... Jul 26 22:16:17.779494345 (58h40m27.502057385s) [upload=56288]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.235612078s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X..X...........X............] duration: 6.170533932s FAIL (2025-07-26 22:16:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.170546906s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X......] duration: 5.767460667s FAIL (2025-07-26 22:16:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767545179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.859636159s FAIL (2025-07-26 22:15:57 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.859674723s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..X...........X...........] duration: 5.894824943s FAIL (2025-07-26 22:15:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.895077306s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [......XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X......] duration: 5.739216909s FAIL (2025-07-26 22:15:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.739234844s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.....] duration: 5.786258707s FAIL (2025-07-26 22:15:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.786279726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234553414s FAIL (2025-07-26 22:15:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== load=56268]: Uploading piece... Jul 26 22:14:18.165005539 (58h38m27.887568579s) [noise=true , address=104.36.82.16:4310 , piece_id=T25OE5YYB7IYZLU27NKMER7KDOQ375FWZ775LXOLJNIMESABDMBQ, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.165246599 (58h38m27.887809639s) [noise=true , address=104.36.82.41:4779 , piece_id=LMQDME257HLSYH75SI2KZFB2O6ZQWBFWDAPSHOOWG7D6IC5CELGQ, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.165475147 (58h38m27.888038187s) [noise=true , address=104.36.82.50:4019 , piece_id=JQBQLO6DCM4WKSTBB6VAC5GMN2N7IBLYYEN5C2ZGREEHSH2YZFSQ, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.165703775 (58h38m27.888266815s) [noise=true , address=104.36.82.55:4215 , piece_id=CMHRYKEHDURE33VOGDT36X5CIGG6CLJIOIBGGXCLFBEHHRDQY6CA, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.165947711 (58h38m27.888510751s) [noise=true , address=104.36.82.46:4514 , piece_id=WEPVZHOHASKTQKK5IFGDTPDJV62GUQFJJKVCBULCGB2NCD6TZQLQ, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.166360161 (58h38m27.888923201s) [noise=true , address=104.36.82.40:4948 , piece_id=TI4CDPBYWCVZIHHBHVK3ZQRWCYTV6FH35LYKQK2DS7EADUT5GDRA, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.166637030 (58h38m27.88920009s) [noise=true , address=104.36.82.36:4628 , piece_id=TT32VXMBJ4SIQZO2VMMRT3QTMWGKBVTN7EVRPEYS7XZ2KBGACZ6Q, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.166891867 (58h38m27.889454908s) [noise=true , address=104.36.82.53:4128 , piece_id=NL7HN5UMBNJ37T7LZHQWDMRWRIHF57CA4YEYV4QR3W5BDLHRZLGA, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.167187012 (58h38m27.889750053s) [noise=true , address=104.36.82.56:4239 , piece_id=RQRIT6Z7HNEGEAJBUCF3AVRRSSEZHVXE4AJPIEAKKIMSP7V5ZFPA, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:18.167428274 (58h38m27.889991304s) [noise=true , address=77.90.31.13:4658 , piece_id=7CKKUG5SPFOWIQOOX3HGO4IKYE5I2RHEJQB5BQ2EO6DJA37JSPSQ, seg_pos={0 0}, upload=56268]: Uploading piece... Jul 26 22:14:19.300903283 (58h38m29.023466353s) [upload=56268]: Next returned nil segment Jul 26 22:14:19.300923121 (58h38m29.023486151s) [upload=56268]: Waiting for error group managing segments... Jul 26 22:14:19.609091824 (58h38m29.331654864s) [noise=true , address=79.127.205.245:4247 , piece_id=K2233FOQGGO6ONPGDLQY63LMY7XAOTVPGZAZFMVQHJ7AY4237ITA, seg_pos={0 0}, upload=56268]: Done uploading piece. err: <nil> Jul 26 22:14:20.215751014 (58h38m29.938314044s) [noise=true , address=77.90.31.13:4658 , piece_id=7CKKUG5SPFOWIQOOX3HGO4IKYE5I2RHEJQB5BQ2EO6DJA37JSPSQ, seg_pos={0 0}, upload=56268]: Done uploading piece. err: <nil> Jul 26 22:14:21.114124803 (58h38m30.836687843s) [noise=true , address=45.91.116.9:4269 , piece_id=KS43RDXOOKS2NQE3YGRC5SH7IAQ46BNA4UOMKJOAXZ645AKXFAAA, seg_pos={0 0}, upload=56268]: Done uploading piece. err: <nil> Jul 26 22:14:28.875509255 (58h38m38.598072295s) [noise=true , address=64.49.13.140:4513 , piece_id=I2EZC453TWVNEC72GYXB4HNPAMRMIO3QW5KF3PM2RPF7D7DEBR4Q, seg_pos={0 0}, upload=56268]: Done uploading piece. err: <nil> Jul 26 22:15:17.068390835 (58h39m26.790953875s) [noise=true , address=104.36.82.5:4872 , piece_id=KYHCROQ5Y2GFYJG6S37UHVCDUPU6HVEHNYCK2CFFIDHJ7Y5R5IYQ, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: write tcp 188.166.225.170:48538->104.36.82.5:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068441222 (58h39m26.791004252s) [noise=true , address=104.36.82.53:4128 , piece_id=NL7HN5UMBNJ37T7LZHQWDMRWRIHF57CA4YEYV4QR3W5BDLHRZLGA, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 188.166.225.170:39302->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068487030 (58h39m26.79105007s) [noise=true , address=104.36.82.56:4239 , piece_id=RQRIT6Z7HNEGEAJBUCF3AVRRSSEZHVXE4AJPIEAKKIMSP7V5ZFPA, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: write tcp 188.166.225.170:47176->104.36.82.56:4239: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068516005 (58h39m26.791079045s) [noise=true , address=104.36.82.15:4292 , piece_id=C4GP4AH7AZLVB3XOWA36X5DVP4I3Y3GY3HRNLG23PJPLZKLE27PQ, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:104.36.82.15:4292): protocol: write tcp 188.166.225.170:43134->104.36.82.15:4292: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068569658 (58h39m26.791132698s) [noise=true , address=104.36.82.41:4779 , piece_id=LMQDME257HLSYH75SI2KZFB2O6ZQWBFWDAPSHOOWG7D6IC5CELGQ, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:1eRovJAp3AyUHP4qkuDvSSGYb9TG6U4tSNzniGJBeYLhkmiUVy, address:104.36.82.41:4779): protocol: write tcp 188.166.225.170:37442->104.36.82.41:4779: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068630464 (58h39m26.791193504s) [noise=true , address=104.36.82.50:4019 , piece_id=JQBQLO6DCM4WKSTBB6VAC5GMN2N7IBLYYEN5C2ZGREEHSH2YZFSQ, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:1ztmvXKmY6hicHZD7iejpcJCKFEhLwemtaYChAp8Po85TRQvGo, address:104.36.82.50:4019): protocol: write tcp 188.166.225.170:44698->104.36.82.50:4019: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068674548 (58h39m26.791237588s) [noise=true , address=104.36.82.55:4215 , piece_id=CMHRYKEHDURE33VOGDT36X5CIGG6CLJIOIBGGXCLFBEHHRDQY6CA, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 188.166.225.170:56978->104.36.82.55:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068722009 (58h39m26.791285049s) [noise=true , address=104.36.82.46:4514 , piece_id=WEPVZHOHASKTQKK5IFGDTPDJV62GUQFJJKVCBULCGB2NCD6TZQLQ, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: write tcp 188.166.225.170:59876->104.36.82.46:4514: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068754411 (58h39m26.791317441s) [noise=true , address=104.36.82.36:4628 , piece_id=TT32VXMBJ4SIQZO2VMMRT3QTMWGKBVTN7EVRPEYS7XZ2KBGACZ6Q, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: write tcp 188.166.225.170:55494->104.36.82.36:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068827751 (58h39m26.791390791s) [noise=true , address=104.36.82.40:4948 , piece_id=TI4CDPBYWCVZIHHBHVK3ZQRWCYTV6FH35LYKQK2DS7EADUT5GDRA, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:12vRsfmHG1SH4gL4RzmiUY9uZyF7uy3UfEWipuH4xaGnQ6NuDZL, address:104.36.82.40:4948): protocol: write tcp 188.166.225.170:38664->104.36.82.40:4948: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068887666 (58h39m26.791450706s) [noise=true , address=104.36.82.9:4028 , piece_id=WABAVIUI7KHUYHQJ56SX5CZSGNB7PCS2D6B7CWSV4A626B6MOXVA, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:12XJfiw7ni2wahnKMgWbVGXF14vU9yzVjEquAA8nJus251cHECV, address:104.36.82.9:4028): protocol: write tcp 188.166.225.170:54962->104.36.82.9:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:17.068960946 (58h39m26.791523986s) [noise=true , address=104.36.82.16:4310 , piece_id=T25OE5YYB7IYZLU27NKMER7KDOQ375FWZ775LXOLJNIMESABDMBQ, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:4310): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:15:17.069023335 (58h39m26.791586376s) [noise=true , address=104.36.82.45:4933 , piece_id=EL47AH3AO2CLKCGD6PWC7MDELRVQTQCMWHX4CFHGYBYNDDJ22BOA, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou, address:104.36.82.45:4933): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:15:17.069142333 (58h39m26.791705373s) [noise=true , address=104.36.82.47:4550 , piece_id=MT7DXE3PX67EWW3FEV5S3HNCSQZKNFNSHOZIKHENJ4NXGBXIFZ4A, seg_pos={0 0}, upload=56268]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:15:17.069154457 (58h39m26.791717487s) [seg_pos={0 0}, upload=56268]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 22:15:17.301086634 (58h39m27.023649684s) [upload=56268]: Done uploading segments... Jul 26 22:15:17.301117223 (58h39m27.023680263s) [upload=56268]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.234566318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.......XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.....] duration: 5.909272563s FAIL (2025-07-26 22:14:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.909294826s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X....] duration: 5.794846294s FAIL (2025-07-26 22:14:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.794873386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.825809591s FAIL (2025-07-26 22:14:48 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.825829267s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X..X...........X..........] duration: 6.26594447s FAIL (2025-07-26 22:14:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.265966202s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [....X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X...] duration: 5.795762554s FAIL (2025-07-26 22:14:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.79579143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [........XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X....] duration: 8.172480159s FAIL (2025-07-26 22:14:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.172508344s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X....X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X..] duration: 21.775898637s FAIL (2025-07-26 22:14:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 21.775922574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234090508s FAIL (2025-07-26 22:14:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== =true , address=104.36.82.21:4501 , piece_id=XPS6SND54Z4XDSXUNGUS6XZZOI5LMFOG5QP73YCW3VGYYYLVHWEQ, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.689051419 (58h37m27.411614459s) [noise=true , address=104.36.82.46:4481 , piece_id=PXGWM6WV3EJVGELLBJKVWBI2UUUQ5VGVRITJLGGLOA245UAMSZOQ, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.689174725 (58h37m27.411737765s) [noise=true , address=79.127.219.51:4201 , piece_id=OVUXTCWDAKN3TSAPEXADIIGY7VBQZJNR6S6TDRLFB5C4X6SJY6TA, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.689311046 (58h37m27.411874086s) [noise=true , address=104.36.82.52:4096 , piece_id=7UL7A7MVGGEBMETMRNG6LIJQLRYYDGKVNJ3NCJ2QVR63M4LXVPAA, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.689480851 (58h37m27.412043881s) [noise=true , address=64.49.13.169:4716 , piece_id=JVSKOZKZ2QFLJ3QRCJUFJYDGFE2IIJG7J6QIYU2TAO5IJOZUAPHQ, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.689739405 (58h37m27.412302435s) [noise=true , address=104.36.82.45:4910 , piece_id=G75DMT7RECF2Q5TTI6KETKTCYYNSZRSBAUDX364DGX7DFWBL2WWQ, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.689868252 (58h37m27.412431292s) [noise=true , address=104.36.82.13:4208 , piece_id=TZRT7XIA3I6G3WFFSDIEY6W2XK7LALHACP3EP3MKCK3J3HYGYTJQ, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.690092100 (58h37m27.41265514s) [noise=true , address=104.36.82.55:4193 , piece_id=V42MNII2IW5LF3X3EREBV3Z2GHD33EBDTMVZFZYFTU3XAK2IXITA, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.690334133 (58h37m27.412897213s) [noise=true , address=104.36.82.56:4238 , piece_id=JCV4CQJQLALJ44ZW6MHJ2VU5YMBSJAKMESHIZAORVMJQRGUJBM2A, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.690730181 (58h37m27.413293221s) [noise=true , address=79.127.201.215:4343 , piece_id=QKQ4Z67QRXJS3D5S4TCW4SDIP43FCFU55X7V7Z7X6MESXUG5NVSA, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.690926025 (58h37m27.413489065s) [noise=true , address=45.91.116.10:4100 , piece_id=L6MURMM65DVHM3WOIZGUCLKCCQYK6SO7XGK65IEQQBZ5I65GCFXA, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.691115307 (58h37m27.413678338s) [noise=true , address=77.90.31.15:4732 , piece_id=DMVTBBWTC2TDUNGIY53Z73KNG3PDP3RLHUNWCM2E65AENAROZNPQ, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.691375646 (58h37m27.413938686s) [noise=true , address=104.36.82.47:4536 , piece_id=OJREYGDFANI5JSK2GUBONHGTX4YVBGU7EYA4SQE3W366DUOL5G4A, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.691605676 (58h37m27.414168716s) [noise=true , address=64.49.13.138:4483 , piece_id=KZTUFFHAFGASMXZZZ4LIYVCOXXRK4OQIXIWCI7YV3J3Z66OZRJJA, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.691795128 (58h37m27.414358169s) [noise=true , address=79.127.219.38:4009 , piece_id=SXD7UDZMLY6N77M3M4D7YIY3U7R73LLFQ7CQ4TNTGFEC2G5YYYLA, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:17.691974091 (58h37m27.414537121s) [noise=true , address=104.36.82.53:4125 , piece_id=VECEYTGCQ2QNJWNCCH6F22CMSFCQT66DJ4UK7XLGJQGJOKZ5JEZQ, seg_pos={0 0}, upload=56254]: Uploading piece... Jul 26 22:13:18.732148128 (58h37m28.454711168s) [noise=true , address=79.127.219.47:4061 , piece_id=XAJOPAFUNYBPM4T4F3P64KB5M43VREUNEBQF34KEBAFTDJYGVHMA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:13:18.796071122 (58h37m28.518634152s) [upload=56254]: Next returned nil segment Jul 26 22:13:18.796085579 (58h37m28.518648619s) [upload=56254]: Waiting for error group managing segments... Jul 26 22:13:19.106955891 (58h37m28.829518921s) [noise=true , address=79.127.205.245:4240 , piece_id=JWK4CFNGQNVRLBDWEXCVRRPCJRRXTISH6OB6RU7J6HVDJWMTU2QA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: <nil> Jul 26 22:13:19.587339794 (58h37m29.309902834s) [noise=true , address=64.49.13.169:4716 , piece_id=JVSKOZKZ2QFLJ3QRCJUFJYDGFE2IIJG7J6QIYU2TAO5IJOZUAPHQ, seg_pos={0 0}, upload=56254]: Done uploading piece. err: <nil> Jul 26 22:13:19.731860387 (58h37m29.454423417s) [noise=true , address=77.90.31.15:4732 , piece_id=DMVTBBWTC2TDUNGIY53Z73KNG3PDP3RLHUNWCM2E65AENAROZNPQ, seg_pos={0 0}, upload=56254]: Done uploading piece. err: <nil> Jul 26 22:13:19.796608079 (58h37m29.519171109s) [noise=true , address=45.91.116.10:4100 , piece_id=L6MURMM65DVHM3WOIZGUCLKCCQYK6SO7XGK65IEQQBZ5I65GCFXA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: <nil> Jul 26 22:13:20.696219534 (58h37m30.418782574s) [noise=true , address=79.127.219.51:4201 , piece_id=OVUXTCWDAKN3TSAPEXADIIGY7VBQZJNR6S6TDRLFB5C4X6SJY6TA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: <nil> Jul 26 22:13:25.965238238 (58h37m35.687801278s) [noise=true , address=79.127.201.215:4343 , piece_id=QKQ4Z67QRXJS3D5S4TCW4SDIP43FCFU55X7V7Z7X6MESXUG5NVSA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: <nil> Jul 26 22:13:27.439521758 (58h37m37.162084798s) [noise=true , address=79.127.219.38:4009 , piece_id=SXD7UDZMLY6N77M3M4D7YIY3U7R73LLFQ7CQ4TNTGFEC2G5YYYLA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: <nil> Jul 26 22:13:29.030707979 (58h37m38.753271019s) [noise=true , address=64.49.13.138:4483 , piece_id=KZTUFFHAFGASMXZZZ4LIYVCOXXRK4OQIXIWCI7YV3J3Z66OZRJJA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: <nil> Jul 26 22:14:16.596068175 (58h38m26.318631225s) [noise=true , address=104.36.82.46:4481 , piece_id=PXGWM6WV3EJVGELLBJKVWBI2UUUQ5VGVRITJLGGLOA245UAMSZOQ, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596096229 (58h38m26.318659269s) [noise=true , address=104.36.82.55:4193 , piece_id=V42MNII2IW5LF3X3EREBV3Z2GHD33EBDTMVZFZYFTU3XAK2IXITA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: rpc: dial tcp 104.36.82.55:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596134392 (58h38m26.318697432s) [noise=true , address=104.36.82.21:4501 , piece_id=XPS6SND54Z4XDSXUNGUS6XZZOI5LMFOG5QP73YCW3VGYYYLVHWEQ, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:1e1R3gjh5CYJdDqck7xcR6A19Hb443FWbX8X2AQikxt5f4AVvU, address:104.36.82.21:4501): protocol: rpc: dial tcp 104.36.82.21:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596147095 (58h38m26.318710136s) [noise=true , address=104.36.82.56:4238 , piece_id=JCV4CQJQLALJ44ZW6MHJ2VU5YMBSJAKMESHIZAORVMJQRGUJBM2A, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: write tcp 188.166.225.170:51732->104.36.82.56:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596187243 (58h38m26.318750283s) [noise=true , address=104.36.82.52:4096 , piece_id=7UL7A7MVGGEBMETMRNG6LIJQLRYYDGKVNJ3NCJ2QVR63M4LXVPAA, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596192913 (58h38m26.318755954s) [noise=true , address=104.36.82.47:4536 , piece_id=OJREYGDFANI5JSK2GUBONHGTX4YVBGU7EYA4SQE3W366DUOL5G4A, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: rpc: dial tcp 104.36.82.47:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596226407 (58h38m26.318789447s) [noise=true , address=104.36.82.45:4910 , piece_id=G75DMT7RECF2Q5TTI6KETKTCYYNSZRSBAUDX364DGX7DFWBL2WWQ, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:12bRYzNggkq7QWFTP9fB94dTrMMX8AYvKMt2zjmJBzT1VwqyPVZ, address:104.36.82.45:4910): protocol: rpc: dial tcp 104.36.82.45:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596230616 (58h38m26.318793656s) [noise=true , address=104.36.82.13:4208 , piece_id=TZRT7XIA3I6G3WFFSDIEY6W2XK7LALHACP3EP3MKCK3J3HYGYTJQ, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:12SXSN3HpeYwgHRpFFST5wwP3hhXEe3QB8VzWTNNE19sFoWW4o5, address:104.36.82.13:4208): protocol: rpc: dial tcp 104.36.82.13:4208: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596263399 (58h38m26.318826439s) [noise=true , address=104.36.82.53:4125 , piece_id=VECEYTGCQ2QNJWNCCH6F22CMSFCQT66DJ4UK7XLGJQGJOKZ5JEZQ, seg_pos={0 0}, upload=56254]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:104.36.82.53:4125): protocol: write tcp 188.166.225.170:50026->104.36.82.53:4125: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:14:16.596295149 (58h38m26.318858189s) [seg_pos={0 0}, upload=56254]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:14:16.827856077 (58h38m26.550419117s) [upload=56254]: Done uploading segments... Jul 26 22:14:16.827882508 (58h38m26.550445548s) [upload=56254]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.23410685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.869448024s FAIL (2025-07-26 22:14:06 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.869471308s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.923547261s FAIL (2025-07-26 22:13:49 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.923584252s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.........XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X...] duration: 5.742564493s FAIL (2025-07-26 22:13:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.742597356s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.240038206s FAIL (2025-07-26 22:13:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== WEOA63RQBFAQ4MLAYIEBKCOI6WAJ23ULBWE5UMGRA, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.214306729 (58h36m26.936869769s) [noise=true , address=104.36.82.47:4550 , piece_id=5UDTFVACVKPOMGUKQW7DZLSTFUGWKJZBFWGB5Z7OAUUZFJAQRIZA, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.214410458 (58h36m26.936973498s) [noise=true , address=79.127.219.38:4004 , piece_id=C5HYQG2VXLBBNYOGFBXZUVOEXZQ46D3E6SMHLAUPLFUAIOGR4DJQ, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.214738075 (58h36m26.937301115s) [noise=true , address=104.36.82.17:4329 , piece_id=NGQLUUTSHH52U2HZY3FVFRCUHMYBEEQPGTLNJHMO5LPOHLP6NHRQ, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.215027750 (58h36m26.93759079s) [noise=true , address=104.36.82.56:4233 , piece_id=ZTQUAJEUCOSLJ3KSWXIQDOV26ECJRG3OBJRUP5AJIDQ3BD7HJ72A, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.215100328 (58h36m26.937663368s) [noise=true , address=77.90.31.16:4743 , piece_id=2C5DBIJOLFM6GRCDNKTS6DFCPJOKOKWT5QACCWLT5ZP2WYCLIJSQ, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.215313155 (58h36m26.937876195s) [noise=true , address=104.36.82.50:4040 , piece_id=JGECBC2AY4KIP2DO6YEDJKKRRUVEY5Z7HKTQAXR2GYQCBYTCRYTQ, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.215565197 (58h36m26.938128238s) [noise=true , address=104.36.82.46:4503 , piece_id=ZHLN7OD3QRUPLKW6CVMVEIDBMT2A2GYUDSJY2YEK4YOAK4Z6K4FA, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.215639259 (58h36m26.93820229s) [noise=true , address=104.36.82.42:4831 , piece_id=N6MP2GLKXSG6EEGC5MLGEN5YZUWM2EAZVXUVUHMJLVCO2JKYSMUQ, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.215854021 (58h36m26.938417061s) [noise=true , address=104.36.82.39:4751 , piece_id=45GZD5U4GW6CRYMRT6KM2PRXU635JAU5DIL2DLQJRNO6LHE423JA, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.216063661 (58h36m26.938626701s) [noise=true , address=77.90.31.10:4540 , piece_id=CR7LCXQC2FG73GQGLD2QM75WDPT6256ZJKDPVX2NFEC4XY25H5LQ, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.216133655 (58h36m26.938696695s) [noise=true , address=156.146.43.225:4449 , piece_id=J3QGIO6VNZT6YQ3E7DRFRPGIYISYXG6GMOYVXTPQRJFOFQGWV2OQ, seg_pos={0 0}, upload=56240]: Uploading piece... Jul 26 22:12:17.484733840 (58h36m27.20729687s) [noise=true , address=77.90.31.10:4540 , piece_id=CR7LCXQC2FG73GQGLD2QM75WDPT6256ZJKDPVX2NFEC4XY25H5LQ, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: write tcp 188.166.225.170:56814->77.90.31.10:4540: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:18.071986786 (58h36m27.794549816s) [noise=true , address=64.49.13.138:4483 , piece_id=KDFO33MKCHTMAVOGTT25QV2ZVZ3K6BHQK2GCHOLCVU3JTR7NTFBQ, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:12:18.363157086 (58h36m28.085720126s) [upload=56240]: Next returned nil segment Jul 26 22:12:18.363179367 (58h36m28.085742407s) [upload=56240]: Waiting for error group managing segments... Jul 26 22:12:18.662483990 (58h36m28.38504703s) [noise=true , address=79.127.205.233:4656 , piece_id=TIWSLAY63DC3GATG5BTJUX4X7GROZRVELDTVDEO43TJ4WWGYOSMA, seg_pos={0 0}, upload=56240]: Done uploading piece. err: <nil> Jul 26 22:12:19.157730280 (58h36m28.88029332s) [noise=true , address=79.127.219.38:4004 , piece_id=C5HYQG2VXLBBNYOGFBXZUVOEXZQ46D3E6SMHLAUPLFUAIOGR4DJQ, seg_pos={0 0}, upload=56240]: Done uploading piece. err: <nil> Jul 26 22:12:19.217984034 (58h36m28.940547104s) [noise=true , address=77.90.31.16:4743 , piece_id=2C5DBIJOLFM6GRCDNKTS6DFCPJOKOKWT5QACCWLT5ZP2WYCLIJSQ, seg_pos={0 0}, upload=56240]: Done uploading piece. err: <nil> Jul 26 22:12:19.826884929 (58h36m29.549447959s) [noise=true , address=156.146.43.225:4449 , piece_id=J3QGIO6VNZT6YQ3E7DRFRPGIYISYXG6GMOYVXTPQRJFOFQGWV2OQ, seg_pos={0 0}, upload=56240]: Done uploading piece. err: <nil> Jul 26 22:13:16.123096312 (58h37m25.845659352s) [noise=true , address=104.36.82.50:4040 , piece_id=JGECBC2AY4KIP2DO6YEDJKKRRUVEY5Z7HKTQAXR2GYQCBYTCRYTQ, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:12gozJwRZvMK7Vbsdy775PtGynixvh3z5MHWwaLxzkrExseZosn, address:104.36.82.50:4040): protocol: rpc: dial tcp 104.36.82.50:4040: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123143161 (58h37m25.845706202s) [noise=true , address=104.36.82.46:4503 , piece_id=ZHLN7OD3QRUPLKW6CVMVEIDBMT2A2GYUDSJY2YEK4YOAK4Z6K4FA, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: rpc: dial tcp 104.36.82.46:4503: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123170594 (58h37m25.845733634s) [noise=true , address=104.36.82.42:4831 , piece_id=N6MP2GLKXSG6EEGC5MLGEN5YZUWM2EAZVXUVUHMJLVCO2JKYSMUQ, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: rpc: dial tcp 104.36.82.42:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123199579 (58h37m25.845762619s) [noise=true , address=104.36.82.53:4124 , piece_id=CKJ3QL4LGLLTFXDJF5NQYPKWPYKJE5JT57SKJS2Q33ATN4EVTY4Q, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: rpc: dial tcp 104.36.82.53:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123226751 (58h37m25.845789781s) [noise=true , address=104.36.82.51:4049 , piece_id=G7BQFMSORNONRNLKKHHO7SBHQ34NVM7IXO3RPATFZZXBVEBGRT4A, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:104.36.82.51:4049): protocol: rpc: dial tcp 104.36.82.51:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123258933 (58h37m25.845821973s) [noise=true , address=104.36.82.54:4159 , piece_id=3ZABUUBFVRBWEOA63RQBFAQ4MLAYIEBKCOI6WAJ23ULBWE5UMGRA, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:104.36.82.54:4159): protocol: rpc: dial tcp 104.36.82.54:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123287938 (58h37m25.845850968s) [noise=true , address=104.36.82.39:4751 , piece_id=45GZD5U4GW6CRYMRT6KM2PRXU635JAU5DIL2DLQJRNO6LHE423JA, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:128StmvpG1KYRXeu3gzEEfRmYZgR5ACJRf2xGZcEiZHsfk1KNyr, address:104.36.82.39:4751): protocol: rpc: dial tcp 104.36.82.39:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123317555 (58h37m25.845880585s) [noise=true , address=104.36.82.2:4748 , piece_id=WR4N72IYGBTSNQQODLNXEBBI7425FPQQXPSOQBRYPFW4XKNHQSFA, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:104.36.82.2:4748): protocol: rpc: dial tcp 104.36.82.2:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123345408 (58h37m25.845908448s) [noise=true , address=104.36.82.17:4329 , piece_id=NGQLUUTSHH52U2HZY3FVFRCUHMYBEEQPGTLNJHMO5LPOHLP6NHRQ, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:104.36.82.17:4329): protocol: rpc: dial tcp 104.36.82.17:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123376027 (58h37m25.845939057s) [noise=true , address=104.36.82.55:4224 , piece_id=CCKAGOHSPAB4EAYMW2UPTT6JKQAXSEYM7FBILTONKF6JAMZCKXNA, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: rpc: dial tcp 104.36.82.55:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123402287 (58h37m25.845965327s) [noise=true , address=104.36.82.47:4550 , piece_id=5UDTFVACVKPOMGUKQW7DZLSTFUGWKJZBFWGB5Z7OAUUZFJAQRIZA, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123433116 (58h37m25.845996156s) [noise=true , address=104.36.82.56:4233 , piece_id=ZTQUAJEUCOSLJ3KSWXIQDOV26ECJRG3OBJRUP5AJIDQ3BD7HJ72A, seg_pos={0 0}, upload=56240]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: write tcp 188.166.225.170:45126->104.36.82.56:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:16.123446001 (58h37m25.846009041s) [seg_pos={0 0}, upload=56240]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 22:13:16.359574488 (58h37m26.082137518s) [upload=56240]: Done uploading segments... Jul 26 22:13:16.359601920 (58h37m26.08216496s) [upload=56240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.240061499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X..X...........X.........] duration: 5.992527739s FAIL (2025-07-26 22:13:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.992550132s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.04312826s FAIL (2025-07-26 22:12:50 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 2.043154639s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X....X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.] duration: 5.71527448s FAIL (2025-07-26 22:12:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.715301751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..X....X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X] duration: 30.229617449s FAIL (2025-07-26 22:12:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.229643438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [..........XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X..] duration: 5.719262978s FAIL (2025-07-26 22:12:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.719289149s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.239258533s FAIL (2025-07-26 22:12:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ADY7ERB6776BZIPTYUHJZBEPYZNQMGCQ2SS5YVWGNRQ, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.801272097 (58h35m21.523835137s) [noise=true , address=64.49.13.137:4500 , piece_id=5XEYXW7PK6AEHPPACHIJJI3BOINFLMS2UAVUO4BALHKGQOKR7QGA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.801414069 (58h35m21.523977099s) [noise=true , address=79.127.219.51:4210 , piece_id=PWFRJK4JGVZRSXBYBKEBBKG4GUSNQ4VHZFMFLZFKJ2CTBT64AL2Q, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.801595646 (58h35m21.524158686s) [noise=true , address=104.36.82.5:4858 , piece_id=OFOJJR3PZWGEQUWRZ5IF5OL555ZKI3KOLADHJS5AQNYTVTCB74YQ, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.801655791 (58h35m21.524218821s) [noise=true , address=79.127.201.215:4359 , piece_id=BAPJCNESFXOPTNYLCGFY2UOQ6TGGRJ6XCVH67ZB23DSVUGUUATSA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.801929054 (58h35m21.524492094s) [noise=true , address=104.36.82.47:4517 , piece_id=JLHP2YTEK7YSUL352SJSDEXSLQDOJZSL2LGJJAYTLGDPL6DPFDIA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.802255709 (58h35m21.524818739s) [noise=true , address=64.49.13.169:4716 , piece_id=J626AHS4IUINN5EVEXY3OAN3PWVEJ5LRJCY3CMGJBEPPBSYVHV7Q, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.802512070 (58h35m21.52507511s) [noise=true , address=79.127.219.38:4017 , piece_id=JV6S5X45YPVX4ROJ3L3NZKVR22REIT3GPSJPO4I6DB7DZB2LMJPA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.802551846 (58h35m21.525114886s) [noise=true , address=104.36.82.54:4168 , piece_id=Q6R46ZTM6N233IQNLCND5UIRXMX65QLP3L5KXYV75PZ7OEFRG5YA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.802796774 (58h35m21.525359814s) [noise=true , address=104.36.82.46:4495 , piece_id=TGNZJZHXH3AHTCICHVIU5RX7O22JGTA43JNCDDO5CXUS6TBZHL6A, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.802897317 (58h35m21.525460347s) [noise=true , address=77.90.31.13:4634 , piece_id=O3M5XN5HFXPMJ7TYCM547J4EC6PRX2YS4CCDPIFE7E6OAHVS6M5Q, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.803035952 (58h35m21.525598992s) [noise=true , address=156.146.43.225:4448 , piece_id=7SIGF7IBQGNVZU4HAEY4VT2CSFO4F4X3AWBWW23DGJYD4HGQUDUQ, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.803265120 (58h35m21.52582815s) [noise=true , address=64.49.13.138:4483 , piece_id=GLNUK3MQG7BSWUT6XLFN7PGM4FZWUW3326KI5W6HOFU732QYVQKA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.803653464 (58h35m21.526216504s) [noise=true , address=104.36.82.52:4111 , piece_id=EMS7N2EKLC2M7QQ3EMCJ23ISGQC3DRJYSKVO6FSF4QOT3ZCC55HA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.803975740 (58h35m21.52653878s) [noise=true , address=104.36.82.56:4264 , piece_id=2O6AGD54EONNKCIZHYV57TIH4VU3N7KKYAEMOK3KP6GOXMZ4M3UA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:11.804687822 (58h35m21.527250862s) [noise=true , address=104.36.82.55:4194 , piece_id=FYU7X3UXHNDHD4NJ7PGGPBTQ2Z4LTLHOULTXCI3YZRY4SRMNA5KA, seg_pos={0 0}, upload=56224]: Uploading piece... Jul 26 22:11:12.913964542 (58h35m22.636527582s) [upload=56224]: Next returned nil segment Jul 26 22:11:12.913981745 (58h35m22.636544785s) [upload=56224]: Waiting for error group managing segments... Jul 26 22:11:13.267601349 (58h35m22.990164389s) [noise=true , address=64.49.13.169:4716 , piece_id=J626AHS4IUINN5EVEXY3OAN3PWVEJ5LRJCY3CMGJBEPPBSYVHV7Q, seg_pos={0 0}, upload=56224]: Done uploading piece. err: <nil> Jul 26 22:11:13.298408683 (58h35m23.020971723s) [noise=true , address=64.49.13.138:4483 , piece_id=GLNUK3MQG7BSWUT6XLFN7PGM4FZWUW3326KI5W6HOFU732QYVQKA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: <nil> Jul 26 22:11:13.741665039 (58h35m23.464228079s) [noise=true , address=77.90.31.13:4634 , piece_id=O3M5XN5HFXPMJ7TYCM547J4EC6PRX2YS4CCDPIFE7E6OAHVS6M5Q, seg_pos={0 0}, upload=56224]: Done uploading piece. err: <nil> Jul 26 22:11:13.812442078 (58h35m23.535005128s) [noise=true , address=79.127.219.51:4210 , piece_id=PWFRJK4JGVZRSXBYBKEBBKG4GUSNQ4VHZFMFLZFKJ2CTBT64AL2Q, seg_pos={0 0}, upload=56224]: Done uploading piece. err: <nil> Jul 26 22:11:14.848066798 (58h35m24.570629838s) [noise=true , address=64.49.13.137:4500 , piece_id=5XEYXW7PK6AEHPPACHIJJI3BOINFLMS2UAVUO4BALHKGQOKR7QGA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: <nil> Jul 26 22:11:16.292975735 (58h35m26.015538785s) [noise=true , address=156.146.43.225:4448 , piece_id=7SIGF7IBQGNVZU4HAEY4VT2CSFO4F4X3AWBWW23DGJYD4HGQUDUQ, seg_pos={0 0}, upload=56224]: Done uploading piece. err: <nil> Jul 26 22:11:17.294887322 (58h35m27.017450362s) [noise=true , address=79.127.219.38:4017 , piece_id=JV6S5X45YPVX4ROJ3L3NZKVR22REIT3GPSJPO4I6DB7DZB2LMJPA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: <nil> Jul 26 22:11:21.313497999 (58h35m31.036061039s) [noise=true , address=79.127.201.215:4359 , piece_id=BAPJCNESFXOPTNYLCGFY2UOQ6TGGRJ6XCVH67ZB23DSVUGUUATSA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: <nil> Jul 26 22:12:10.705742718 (58h36m20.428305748s) [noise=true , address=104.36.82.52:4111 , piece_id=EMS7N2EKLC2M7QQ3EMCJ23ISGQC3DRJYSKVO6FSF4QOT3ZCC55HA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: rpc: dial tcp 104.36.82.52:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705785851 (58h36m20.428348891s) [noise=true , address=104.36.82.47:4517 , piece_id=JLHP2YTEK7YSUL352SJSDEXSLQDOJZSL2LGJJAYTLGDPL6DPFDIA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705839654 (58h36m20.428402684s) [noise=true , address=104.36.82.5:4858 , piece_id=OFOJJR3PZWGEQUWRZ5IF5OL555ZKI3KOLADHJS5AQNYTVTCB74YQ, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:12PasA9NPExHCQgMSK1E1kz2HHkpXQmWq7LmrwY6sRmf22avri, address:104.36.82.5:4858): protocol: write tcp 188.166.225.170:33292->104.36.82.5:4858: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705882535 (58h36m20.428445565s) [noise=true , address=104.36.82.37:4655 , piece_id=KKBHHAEX2ADY7ERB6776BZIPTYUHJZBEPYZNQMGCQ2SS5YVWGNRQ, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:104.36.82.37:4655): protocol: rpc: dial tcp 104.36.82.37:4655: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705892755 (58h36m20.428455796s) [noise=true , address=104.36.82.56:4264 , piece_id=2O6AGD54EONNKCIZHYV57TIH4VU3N7KKYAEMOK3KP6GOXMZ4M3UA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705912202 (58h36m20.428475242s) [noise=true , address=104.36.82.54:4168 , piece_id=Q6R46ZTM6N233IQNLCND5UIRXMX65QLP3L5KXYV75PZ7OEFRG5YA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: rpc: dial tcp 104.36.82.54:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705925408 (58h36m20.428488448s) [noise=true , address=104.36.82.55:4194 , piece_id=FYU7X3UXHNDHD4NJ7PGGPBTQ2Z4LTLHOULTXCI3YZRY4SRMNA5KA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: write tcp 188.166.225.170:60730->104.36.82.55:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705941418 (58h36m20.428504458s) [noise=true , address=104.36.82.46:4495 , piece_id=TGNZJZHXH3AHTCICHVIU5RX7O22JGTA43JNCDDO5CXUS6TBZHL6A, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:1YpHarCJ3ouMGTs59udhU7az9h68ELEHTa2aTAax2chR2Gux1x, address:104.36.82.46:4495): protocol: rpc: dial tcp 104.36.82.46:4495: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705951347 (58h36m20.428514388s) [noise=true , address=104.36.82.11:4126 , piece_id=BX3PBN2TQ4PDBVJ4YJ3FSPFL5TJT5DDKORAUJYJ7W2TQNRPW3EVA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:12R3SVXrhJsmYTXR4vDSkfaiqDRox21u17SLmQc9sEab1B7zZRr, address:104.36.82.11:4126): protocol: rpc: dial tcp 104.36.82.11:4126: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705970564 (58h36m20.428533604s) [noise=true , address=104.36.82.9:4018 , piece_id=NVUCS5UFJN4UODILW577J33OSQI3GEMKIXRQQO3AME7XUYGE6VVA, seg_pos={0 0}, upload=56224]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:104.36.82.9:4018): protocol: rpc: dial tcp 104.36.82.9:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:10.705987556 (58h36m20.428550596s) [seg_pos={0 0}, upload=56224]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:12:10.943696106 (58h36m20.666259136s) [upload=56224]: Done uploading segments... Jul 26 22:12:10.943730513 (58h36m20.666293553s) [upload=56224]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.239269143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X..X...........X........] duration: 5.879998297s FAIL (2025-07-26 22:12:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.880018596s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.865778804s FAIL (2025-07-26 22:11:58 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.865808429s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [...........XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.] duration: 5.751403393s FAIL (2025-07-26 22:11:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.751436266s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.861783307s FAIL (2025-07-26 22:11:22 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.861819397s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)