Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X..X...........X........................] duration: 5.890114637s FAIL (2025-07-26 22:24:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.890138111s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.227968285s FAIL (2025-07-26 22:24:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 0}, upload=56420]: Done begin upload segment. Jul 26 22:23:25.972398552 (58h47m35.694961592s) [noise=true , address=104.36.82.41:4777 , piece_id=ZJNTVVF4GHHZRD4FVV4JK54L2LCXZM23RTO772NZDERHVLJ76M3A, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.972444349 (58h47m35.695007389s) [noise=true , address=79.127.201.215:4343 , piece_id=XIU6EYPQPAX56W5Q2LRQYTOGHTD2OCCCBBZUKEUW5HXER4TE3A5A, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.972756186 (58h47m35.695319216s) [noise=true , address=104.36.82.11:4141 , piece_id=4BFXJWB2U6AIMQPGOOVHIHXRGJXFQ7U7DQMPOFGZU2YEPZNCEDXQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.972979143 (58h47m35.695542183s) [noise=true , address=79.127.205.235:4681 , piece_id=YUBQM2UVOERNKISW245NL4VVCSMJOA7NG4Y5Z5AEGES4LVJV6VBA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973212639 (58h47m35.695775689s) [noise=true , address=104.36.82.56:4232 , piece_id=QRNYYOAT4ROWBIWLSVNJBX6ILNETOH7ZWC6NT4ZXMVKUZG55PAEA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973244049 (58h47m35.695807079s) [noise=true , address=77.90.31.21:4897 , piece_id=5UW5IFKFZSLZ43RALU6HAG27O5PNDMROINANBJGP6SXJHMWKLCOQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973465202 (58h47m35.696028242s) [noise=true , address=104.36.82.40:4961 , piece_id=IXV2DTBXEFMRBXYAOBNJ6DMWGW5UTMGCLT2T2ZF63YK5ZXQHNCSA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973697587 (58h47m35.696260627s) [noise=true , address=104.36.82.21:4477 , piece_id=MRTHYSAXYPUA24FNBAIFZBF6RHQRSIAQMOUDKJFELXDMZDEDBMNQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973829670 (58h47m35.69639271s) [noise=true , address=104.36.82.55:4228 , piece_id=SJKIDFGXTGRU2NVAKQOQLZI5UTR23N3FL4C4BEJJOWGWKV5SKGSA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973934961 (58h47m35.696497991s) [noise=true , address=79.127.205.245:4236 , piece_id=6VYHJKM5QWRINFV3UPZA5XF2APU5PTETTGEB7SAZ7JGYC4MR4Y6A, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974136507 (58h47m35.696699547s) [noise=true , address=79.127.219.38:4006 , piece_id=JQKKKTRJJ74YCW5IHRHHSI2GHOJ6OMRHKOLU5KMA7PACW3QUGPKQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974262087 (58h47m35.696825127s) [noise=true , address=79.127.220.101:4048 , piece_id=2QIFNH54UYSCNPWBRA7PQW37VQUJFBPFQNLGV7QEEKBEZIC3YSKA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974329597 (58h47m35.696892637s) [noise=true , address=45.91.116.14:4206 , piece_id=RJTL5VWYHRAQFCS52V2TXX2YI6PCS4ICDFPXFLE7YY2JHFNLU2RA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974528126 (58h47m35.697091166s) [noise=true , address=79.127.219.51:4212 , piece_id=IBFWKL45TRM4Z4NJZP5JAKUIXAUF56QISVZG4IFSAQ3GTYYBK6DQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974670969 (58h47m35.697234009s) [noise=true , address=79.127.205.233:4645 , piece_id=JJV7Y4NIPOODJZSBCAWRY2DOJ7OAOME7NGG5U6OSSMKU4K6B5GCQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974725443 (58h47m35.697288484s) [noise=true , address=79.127.205.234:4620 , piece_id=EZEJUEN54TU4NRODEFHGR7RMICQWWWQOZJKNFQCT3SVPTSNMJDQA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974998145 (58h47m35.697561175s) [noise=true , address=45.91.116.10:4100 , piece_id=MIKZKGUB45NZMUXOIQPR4WTDIJA3F3MY7PMLXK67ZM6A77OBMUKA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.976674393 (58h47m35.699237433s) [noise=true , address=104.36.82.36:4626 , piece_id=TGCZ7USQJ4SSKA7MPLXQ43I3RCALKY2QREL3H6F6ES3LHGYG26PA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:26.968452678 (58h47m36.691015718s) [noise=true , address=79.127.220.101:4048 , piece_id=2QIFNH54UYSCNPWBRA7PQW37VQUJFBPFQNLGV7QEEKBEZIC3YSKA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:23:27.068659100 (58h47m36.79122214s) [upload=56420]: Next returned nil segment Jul 26 22:23:27.068676964 (58h47m36.791240004s) [upload=56420]: Waiting for error group managing segments... Jul 26 22:23:27.358757576 (58h47m37.081320616s) [noise=true , address=79.127.205.245:4236 , piece_id=6VYHJKM5QWRINFV3UPZA5XF2APU5PTETTGEB7SAZ7JGYC4MR4Y6A, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.378592004 (58h47m37.101155034s) [noise=true , address=79.127.205.235:4681 , piece_id=YUBQM2UVOERNKISW245NL4VVCSMJOA7NG4Y5Z5AEGES4LVJV6VBA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.392488451 (58h47m37.115051481s) [noise=true , address=79.127.205.234:4620 , piece_id=EZEJUEN54TU4NRODEFHGR7RMICQWWWQOZJKNFQCT3SVPTSNMJDQA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.473674272 (58h47m37.196237312s) [noise=true , address=79.127.205.233:4645 , piece_id=JJV7Y4NIPOODJZSBCAWRY2DOJ7OAOME7NGG5U6OSSMKU4K6B5GCQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.968271128 (58h47m37.690834168s) [noise=true , address=79.127.219.38:4006 , piece_id=JQKKKTRJJ74YCW5IHRHHSI2GHOJ6OMRHKOLU5KMA7PACW3QUGPKQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.989648718 (58h47m37.712211748s) [noise=true , address=77.90.31.21:4897 , piece_id=5UW5IFKFZSLZ43RALU6HAG27O5PNDMROINANBJGP6SXJHMWKLCOQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:28.374687590 (58h47m38.09725063s) [noise=true , address=45.91.116.14:4206 , piece_id=RJTL5VWYHRAQFCS52V2TXX2YI6PCS4ICDFPXFLE7YY2JHFNLU2RA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:28.794267918 (58h47m38.516830958s) [noise=true , address=79.127.219.51:4212 , piece_id=IBFWKL45TRM4Z4NJZP5JAKUIXAUF56QISVZG4IFSAQ3GTYYBK6DQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:29.132833164 (58h47m38.855396194s) [noise=true , address=104.36.82.55:4228 , piece_id=SJKIDFGXTGRU2NVAKQOQLZI5UTR23N3FL4C4BEJJOWGWKV5SKGSA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:23:33.270854901 (58h47m42.993417941s) [noise=true , address=45.91.116.10:4100 , piece_id=MIKZKGUB45NZMUXOIQPR4WTDIJA3F3MY7PMLXK67ZM6A77OBMUKA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:40.548138829 (58h47m50.270701869s) [noise=true , address=79.127.201.215:4343 , piece_id=XIU6EYPQPAX56W5Q2LRQYTOGHTD2OCCCBBZUKEUW5HXER4TE3A5A, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:24:24.890240005 (58h48m34.612803045s) [noise=true , address=104.36.82.11:4141 , piece_id=4BFXJWB2U6AIMQPGOOVHIHXRGJXFQ7U7DQMPOFGZU2YEPZNCEDXQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:4141): protocol: write tcp 188.166.225.170:56062->104.36.82.11:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890415782 (58h48m34.612978822s) [noise=true , address=104.36.82.56:4232 , piece_id=QRNYYOAT4ROWBIWLSVNJBX6ILNETOH7ZWC6NT4ZXMVKUZG55PAEA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:1knYU6KSZJCmdpXTRHXZmCsRJYfNGGQCLPEPkWkBgUdyvtSgJe, address:104.36.82.56:4232): protocol: rpc: dial tcp 104.36.82.56:4232: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890459865 (58h48m34.613022895s) [noise=true , address=104.36.82.41:4777 , piece_id=ZJNTVVF4GHHZRD4FVV4JK54L2LCXZM23RTO772NZDERHVLJ76M3A, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt, address:104.36.82.41:4777): protocol: rpc: dial tcp 104.36.82.41:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890465687 (58h48m34.613028727s) [noise=true , address=104.36.82.21:4477 , piece_id=MRTHYSAXYPUA24FNBAIFZBF6RHQRSIAQMOUDKJFELXDMZDEDBMNQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY, address:104.36.82.21:4477): protocol: rpc: dial tcp 104.36.82.21:4477: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890493641 (58h48m34.613056671s) [noise=true , address=104.36.82.40:4961 , piece_id=IXV2DTBXEFMRBXYAOBNJ6DMWGW5UTMGCLT2T2ZF63YK5ZXQHNCSA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:12ZGPLhM6NsdjKLkLCCLSeaNS8ikfnHPZxc3CCw9mLzSznbEoUa, address:104.36.82.40:4961): protocol: rpc: dial tcp 104.36.82.40:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890502517 (58h48m34.613065558s) [noise=true , address=104.36.82.36:4626 , piece_id=TGCZ7USQJ4SSKA7MPLXQ43I3RCALKY2QREL3H6F6ES3LHGYG26PA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: write tcp 188.166.225.170:59550->104.36.82.36:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890523838 (58h48m34.613086879s) [seg_pos={0 0}, upload=56420]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:24:25.117057758 (58h48m34.839620798s) [upload=56420]: Done uploading segments... Jul 26 22:24:25.117085902 (58h48m34.839648932s) [upload=56420]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.22798132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [......XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.................] duration: 5.703097345s FAIL (2025-07-26 22:24:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.703127112s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..X...........X.......................] duration: 6.127793513s FAIL (2025-07-26 22:24:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.127814914s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X..................] duration: 5.718213786s FAIL (2025-07-26 22:24:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.718245947s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 52.870611015s FAIL (2025-07-26 22:23:55 UTC) orbiter warning: operation error S3: GetObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1855ED318630ED12, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: unable to create order limits) Duration: 52.870632677s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X..X...........X......................] duration: 5.994241639s FAIL (2025-07-26 22:23:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.994264023s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232693437s FAIL (2025-07-26 22:23:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== ece... Jul 26 22:22:25.537397386 (58h46m35.259960416s) [noise=true , address=104.36.82.5:4853 , piece_id=O7IZCW4OQS3MUYEIF4WSTO3BQMUMB5GE4VZXM5UJPVWALIANWZGA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537441770 (58h46m35.2600048s) [noise=true , address=104.36.82.10:4090 , piece_id=PBR5CVYTAS26E3UINTYNBMZVYMTH2RI7CCJ5BMNLARD2RYHDHAAA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537643467 (58h46m35.260206507s) [noise=true , address=79.127.219.51:4212 , piece_id=MA3DJ6ORXGWD2TWD6KWAWPUWVI272SP2RONGKO72YOS7WTLRX2CA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537745431 (58h46m35.260308471s) [noise=true , address=45.91.116.9:4267 , piece_id=BXJGR6J2WRGNVIUWWYUPHT74JP2VCS5Z65IFBEUFBHUHWELJQQRA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537917170 (58h46m35.26048021s) [noise=true , address=79.127.219.38:4001 , piece_id=25V3MZW5X5TX722LPWN3YSBQ6YVJQOSLJ2C5B3E5CLFB2VP5AX4Q, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537997574 (58h46m35.260560614s) [noise=true , address=156.146.43.225:4434 , piece_id=ZHPR5WUPHHYOHYUSRBD6JSJA7EPHLIA4JG5YORE4FAAVKZ5JBIEA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.538240909 (58h46m35.260803949s) [noise=true , address=79.127.219.47:4069 , piece_id=KBJKWO6IYEJ73N4QSAPAUXXI2QDDNRLLEOGBQTJWCWBNPXEVEVMA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.538516717 (58h46m35.261079757s) [noise=true , address=104.36.82.11:4147 , piece_id=DRDND4FEV2JRCUWXLP2T27QC7XPSI3CIIU6UPT2ULQA7B2R5XJYQ, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.538739574 (58h46m35.261302614s) [noise=true , address=104.36.82.12:4157 , piece_id=PO2M57BMNDE7EF6IJAZL7MKRXZH6T5SE4FF467246QEZZ7WDARAA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.538966408 (58h46m35.261529448s) [noise=true , address=79.127.201.215:4348 , piece_id=Z4HWXKKGEMXSBJCSUTQNXTNI2RQXETSGOT44GQKESS5IWIPMOL5Q, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.539204122 (58h46m35.261767152s) [noise=true , address=104.36.82.7:4916 , piece_id=WNIN4YLBIRVVRBI7IEHVEK746GAOZQTFUQUF35ZQ6BGWCONH3SKQ, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.539412511 (58h46m35.261975551s) [noise=true , address=77.90.31.17:4801 , piece_id=K4GQKHZW22LPAGOJLBYU5LWNCBENH4U6XBKWVKATVRYTOOP5VSFQ, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.539673731 (58h46m35.262236761s) [noise=true , address=104.36.82.42:4809 , piece_id=IC3LJZFPUS67I4NXXRMIPLHCPZ4JL3YFKJS2GQS6NRUA2Y2XOFLA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:27.057813139 (58h46m36.780376179s) [upload=56405]: Next returned nil segment Jul 26 22:22:27.057833196 (58h46m36.780396236s) [upload=56405]: Waiting for error group managing segments... Jul 26 22:22:27.540530195 (58h46m37.263093235s) [noise=true , address=77.90.31.16:4759 , piece_id=FVGECBP3ZH7RT5IRD3B3TPZDW3OOTPEWMEEOBI2P2LQBTVDY3HRA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:27.734198634 (58h46m37.456761674s) [noise=true , address=79.127.219.38:4001 , piece_id=25V3MZW5X5TX722LPWN3YSBQ6YVJQOSLJ2C5B3E5CLFB2VP5AX4Q, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:28.594129421 (58h46m38.316692561s) [noise=true , address=45.91.116.9:4267 , piece_id=BXJGR6J2WRGNVIUWWYUPHT74JP2VCS5Z65IFBEUFBHUHWELJQQRA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:22:29.983982629 (58h46m39.706545669s) [noise=true , address=156.146.43.225:4434 , piece_id=ZHPR5WUPHHYOHYUSRBD6JSJA7EPHLIA4JG5YORE4FAAVKZ5JBIEA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:30.754303760 (58h46m40.47686685s) [noise=true , address=79.127.219.51:4212 , piece_id=MA3DJ6ORXGWD2TWD6KWAWPUWVI272SP2RONGKO72YOS7WTLRX2CA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:31.106372341 (58h46m40.828935411s) [noise=true , address=79.127.219.47:4069 , piece_id=KBJKWO6IYEJ73N4QSAPAUXXI2QDDNRLLEOGBQTJWCWBNPXEVEVMA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:33.021379208 (58h46m42.743942248s) [noise=true , address=79.127.201.215:4348 , piece_id=Z4HWXKKGEMXSBJCSUTQNXTNI2RQXETSGOT44GQKESS5IWIPMOL5Q, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:23:24.431394761 (58h47m34.153957801s) [noise=true , address=77.90.31.17:4801 , piece_id=K4GQKHZW22LPAGOJLBYU5LWNCBENH4U6XBKWVKATVRYTOOP5VSFQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: write tcp 188.166.225.170:48086->77.90.31.17:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431460417 (58h47m34.154023457s) [noise=true , address=104.36.82.42:4809 , piece_id=IC3LJZFPUS67I4NXXRMIPLHCPZ4JL3YFKJS2GQS6NRUA2Y2XOFLA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1VZgucaa2gRuFhNxARMb7BDS6EGZXKChGPppbFksbPc5we7fTQ, address:104.36.82.42:4809): protocol: write tcp 188.166.225.170:53788->104.36.82.42:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431521283 (58h47m34.154084353s) [noise=true , address=104.36.82.10:4090 , piece_id=PBR5CVYTAS26E3UINTYNBMZVYMTH2RI7CCJ5BMNLARD2RYHDHAAA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:104.36.82.10:4090): protocol: rpc: dial tcp 104.36.82.10:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431557733 (58h47m34.154120773s) [noise=true , address=104.36.82.11:4147 , piece_id=DRDND4FEV2JRCUWXLP2T27QC7XPSI3CIIU6UPT2ULQA7B2R5XJYQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12nNFCM4wDSvGMeqF8MF6csgKDUV6QyozoC5fZwUaj7MN1Woz4P, address:104.36.82.11:4147): protocol: rpc: dial tcp 104.36.82.11:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431604593 (58h47m34.154167623s) [noise=true , address=104.36.82.7:4916 , piece_id=WNIN4YLBIRVVRBI7IEHVEK746GAOZQTFUQUF35ZQ6BGWCONH3SKQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12fifEPABGWmVzxBHNzhjh3r4ue3RRSeNdydifpXruXtH4FELBK, address:104.36.82.7:4916): protocol: write tcp 188.166.225.170:57742->104.36.82.7:4916: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431639609 (58h47m34.154202649s) [noise=true , address=104.36.82.4:4823 , piece_id=XHXVZHJYWWYHMD7DHWYBUC7XD6MUA2FCU5Z5AX4GGBHEEAWZTMUQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1JWhB5QxbZsTD863tEfkEN1CUbKBMQvMJ9ZaiMAwKvxxXyDkEq, address:104.36.82.4:4823): protocol: rpc: dial tcp 104.36.82.4:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431686670 (58h47m34.15424977s) [noise=true , address=104.36.82.12:4172 , piece_id=QG6U6GZZM4LBGIDQMUD3V5H5URA3KY2UJ6CV52I74WYLMKV52I6A, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1xfh8U6X8B8bpXvyX7JHtAGY6zsApfjQtZghScuo7BerzDj4yj, address:104.36.82.12:4172): protocol: rpc: dial tcp 104.36.82.12:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431743017 (58h47m34.154306057s) [noise=true , address=104.36.82.5:4853 , piece_id=O7IZCW4OQS3MUYEIF4WSTO3BQMUMB5GE4VZXM5UJPVWALIANWZGA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: rpc: dial tcp 104.36.82.5:4853: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431780328 (58h47m34.154343368s) [noise=true , address=104.36.82.35:4614 , piece_id=T652T54FRTWGCMDBKIC2N5FB6ORHUVOF7O35KJMPTSBO4T2OBRRA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4, address:104.36.82.35:4614): protocol: rpc: dial tcp 104.36.82.35:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431823321 (58h47m34.154386351s) [noise=true , address=104.36.82.10:4089 , piece_id=EEUWHIRJ4CPXUMXQW4YRVV7SGTHAIPK77WDSJ54B4EGQEZ5K6UMQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 188.166.225.170:56840->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431868418 (58h47m34.154431448s) [noise=true , address=104.36.82.12:4157 , piece_id=PO2M57BMNDE7EF6IJAZL7MKRXZH6T5SE4FF467246QEZZ7WDARAA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12vPyYh3yrhc3zt4nd5C6TjHGaMAHGcpVwLTcgYUqCKGhAvicsX, address:104.36.82.12:4157): protocol: write tcp 188.166.225.170:35432->104.36.82.12:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431901711 (58h47m34.154464741s) [seg_pos={0 0}, upload=56405]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:23:24.662008447 (58h47m34.384571487s) [upload=56405]: Done uploading segments... Jul 26 22:23:24.662037873 (58h47m34.384600923s) [upload=56405]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.23272148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [.......XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X................] duration: 5.724627013s FAIL (2025-07-26 22:23:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.724652851s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..X...........X.....................] duration: 6.14984238s FAIL (2025-07-26 22:23:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.149866817s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.................] duration: 5.712712734s FAIL (2025-07-26 22:22:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.712737241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X................] duration: 5.769148848s FAIL (2025-07-26 22:22:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.769175578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.242022119s FAIL (2025-07-26 22:22:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== s={0 0}, upload=56390]: Begin upload segment... Jul 26 22:21:25.068506885 (58h45m34.791069925s) [noise=true , address=77.90.31.12:4593 , piece_id=KC74P7E7FVZL2PMY3AEMRBN35MCI5ZJFWC6ZAS5S543SZDWBRSTQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.068903705 (58h45m34.791466745s) [noise=true , address=45.91.116.13:4175 , piece_id=PGQDBZEYXYC4VU5CSUQ67QTNB6E7OOLF5RDLA55WBMRQLI3ZQ42Q, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.069146931 (58h45m34.791709971s) [noise=true , address=79.127.201.215:4353 , piece_id=Q7KIRRUZLZ2HP2ELQQWLEYAZALBALST2CWVUWTXG7VQMGPG4ZCRA, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.069393451 (58h45m34.791956481s) [noise=true , address=45.91.116.8:4065 , piece_id=TO2WL2L2C5EOPJI2AJHDCCYHHLKVCCQBRHYYRES6TEF3C33WEFGQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.069910300 (58h45m34.792473341s) [seg_pos={0 0}, upload=56390]: Done begin upload segment. Jul 26 22:21:25.069982980 (58h45m34.79254602s) [noise=true , address=64.49.13.170:4718 , piece_id=TGKPT5Q4G7JQ7CHGYYDMGUQRMKUKUBDIGNOZV5FEKTPSBIJVOFEQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.070476404 (58h45m34.793039444s) [noise=true , address=79.127.219.47:4061 , piece_id=K4XC6LEKEIC4K77OOJPMKHLN52QWQ4S2ABO6NRL6BQVMGRWDN6RQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.070793881 (58h45m34.793356921s) [noise=true , address=193.149.164.143:4702 , piece_id=6NDU6OTW7FBTRKZWAH5EQPDWDNV6Y2UWT4JXNQGQIS7HMISDC52Q, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.071044641 (58h45m34.793607681s) [noise=true , address=64.49.13.140:4508 , piece_id=JOZGRYT2HT23H4J6255PVRBRVPW65JUQ32D35QILPVJ5TBSNMRRQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.071302925 (58h45m34.793865965s) [noise=true , address=77.90.31.16:4760 , piece_id=VHOGZM2FF4OY64RMV4RVLDCFVQPGNE3HIK3YTEWVBVBQ554N7YTA, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.071542153 (58h45m34.794105193s) [noise=true , address=104.36.82.34:4573 , piece_id=CFHN262XLP74VYQGXWBF2LXAFQM3KWTHNUU736DFQ26ICZW7CTOA, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.071761954 (58h45m34.794324994s) [noise=true , address=104.36.82.21:4503 , piece_id=ZGP3MRWGMWFTYE6LCCN2SBLP3CIOW6IBV6MX2ZMYCZYQECZEWRIA, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.071977105 (58h45m34.794540145s) [noise=true , address=104.36.82.40:4953 , piece_id=55WZF6G656WBDH6GYZEIV5NS636KLXNWWF47UOZUNZCTBSMOLUGQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.072195382 (58h45m34.794758422s) [noise=true , address=104.36.82.43:4838 , piece_id=7IJV4BGZJQEOC2ZAQH5FASVFALOBSMHRUYHMJXZF6UKFX52ZZ4MQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.072458025 (58h45m34.795021065s) [noise=true , address=64.49.13.137:4500 , piece_id=PGWB7M77XZC7WZYAE35XIWPKX5WH7GWO6DHWWGH7PNA6FR4RAGZQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.072652367 (58h45m34.795215407s) [noise=true , address=104.36.82.41:4768 , piece_id=QDOTZ5DZI3WQYGCT4DKVLY6VHXFDTYCQWCF4JFLVRZINWG6QMCZA, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.072845376 (58h45m34.795408416s) [noise=true , address=79.127.205.245:4247 , piece_id=EEG65LOJEVWFOLE6IVSE5JRVCELKHUKOSB6P2Z2TQ5HWTRFRLLSQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.073042744 (58h45m34.795605784s) [noise=true , address=79.127.219.51:4212 , piece_id=GIORRY2MAHL6JFWEOZN5CUZWTQYKI77R2AOPWM4LOQI7FLO7GTMQ, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:25.073231235 (58h45m34.795794275s) [noise=true , address=45.91.116.14:4206 , piece_id=FIOYKDRYBSCQ44XMCYLL6RZUQ2AZTOHLSD3DFT4XJCIFSBONJQFA, seg_pos={0 0}, upload=56390]: Uploading piece... Jul 26 22:21:26.174046067 (58h45m35.896609107s) [upload=56390]: Next returned nil segment Jul 26 22:21:26.174063410 (58h45m35.89662645s) [upload=56390]: Waiting for error group managing segments... Jul 26 22:21:26.495684487 (58h45m36.218247527s) [noise=true , address=79.127.205.245:4247 , piece_id=EEG65LOJEVWFOLE6IVSE5JRVCELKHUKOSB6P2Z2TQ5HWTRFRLLSQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:26.534143549 (58h45m36.256706579s) [noise=true , address=193.149.164.143:4702 , piece_id=6NDU6OTW7FBTRKZWAH5EQPDWDNV6Y2UWT4JXNQGQIS7HMISDC52Q, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:26.538631155 (58h45m36.261194195s) [noise=true , address=64.49.13.137:4500 , piece_id=PGWB7M77XZC7WZYAE35XIWPKX5WH7GWO6DHWWGH7PNA6FR4RAGZQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:27.056903757 (58h45m36.779466797s) [noise=true , address=77.90.31.16:4760 , piece_id=VHOGZM2FF4OY64RMV4RVLDCFVQPGNE3HIK3YTEWVBVBQ554N7YTA, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:27.099218215 (58h45m36.821781255s) [noise=true , address=45.91.116.13:4175 , piece_id=PGQDBZEYXYC4VU5CSUQ67QTNB6E7OOLF5RDLA55WBMRQLI3ZQ42Q, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:28.098191757 (58h45m37.820754797s) [noise=true , address=79.127.219.47:4061 , piece_id=K4XC6LEKEIC4K77OOJPMKHLN52QWQ4S2ABO6NRL6BQVMGRWDN6RQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:28.301650728 (58h45m38.024213768s) [noise=true , address=64.49.13.140:4508 , piece_id=JOZGRYT2HT23H4J6255PVRBRVPW65JUQ32D35QILPVJ5TBSNMRRQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:29.319113115 (58h45m39.041676155s) [noise=true , address=64.49.13.170:4718 , piece_id=TGKPT5Q4G7JQ7CHGYYDMGUQRMKUKUBDIGNOZV5FEKTPSBIJVOFEQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: ecclient: upload failed (node:12AcLsSmVz9sRtuDLs47WoWYC66YSbr2aXBjzBdjayg22daQ2Hp, address:64.49.13.170:4718): protocol: storage node overloaded, request limit: 2000 Jul 26 22:21:29.628868740 (58h45m39.35143178s) [noise=true , address=79.127.219.51:4212 , piece_id=GIORRY2MAHL6JFWEOZN5CUZWTQYKI77R2AOPWM4LOQI7FLO7GTMQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:29.669140119 (58h45m39.391703159s) [noise=true , address=77.90.31.12:4593 , piece_id=KC74P7E7FVZL2PMY3AEMRBN35MCI5ZJFWC6ZAS5S543SZDWBRSTQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:32.510168321 (58h45m42.232731361s) [noise=true , address=45.91.116.14:4206 , piece_id=FIOYKDRYBSCQ44XMCYLL6RZUQ2AZTOHLSD3DFT4XJCIFSBONJQFA, seg_pos={0 0}, upload=56390]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:21:40.493417641 (58h45m50.215980671s) [noise=true , address=79.127.201.215:4353 , piece_id=Q7KIRRUZLZ2HP2ELQQWLEYAZALBALST2CWVUWTXG7VQMGPG4ZCRA, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:21:45.916202462 (58h45m55.638765503s) [noise=true , address=45.91.116.8:4065 , piece_id=TO2WL2L2C5EOPJI2AJHDCCYHHLKVCCQBRHYYRES6TEF3C33WEFGQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: <nil> Jul 26 22:22:23.962289005 (58h46m33.684852085s) [noise=true , address=104.36.82.41:4768 , piece_id=QDOTZ5DZI3WQYGCT4DKVLY6VHXFDTYCQWCF4JFLVRZINWG6QMCZA, seg_pos={0 0}, upload=56390]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: write tcp 188.166.225.170:54930->104.36.82.41:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:23.962372285 (58h46m33.684935325s) [noise=true , address=104.36.82.34:4573 , piece_id=CFHN262XLP74VYQGXWBF2LXAFQM3KWTHNUU736DFQ26ICZW7CTOA, seg_pos={0 0}, upload=56390]: Done uploading piece. err: ecclient: upload failed (node:1yD3c1tEtGt2yYEyzDX5GugUChtkCQa9VTWkBH9ftgwvqn72SU, address:104.36.82.34:4573): protocol: write tcp 188.166.225.170:57110->104.36.82.34:4573: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:23.962382995 (58h46m33.684946035s) [noise=true , address=104.36.82.40:4953 , piece_id=55WZF6G656WBDH6GYZEIV5NS636KLXNWWF47UOZUNZCTBSMOLUGQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: ecclient: upload failed (node:12dhNW98YkWKWzXBnwmTCvzaw9CtvxRW8ELJnAjHnATFhpM1oKn, address:104.36.82.40:4953): protocol: write tcp 188.166.225.170:42892->104.36.82.40:4953: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:23.962428833 (58h46m33.684991923s) [noise=true , address=104.36.82.21:4503 , piece_id=ZGP3MRWGMWFTYE6LCCN2SBLP3CIOW6IBV6MX2ZMYCZYQECZEWRIA, seg_pos={0 0}, upload=56390]: Done uploading piece. err: ecclient: upload failed (node:17F3Gn5LYgekp8WgagEdFXtRu8jCc27qA9UuW7pWsJ9FUFKFSi, address:104.36.82.21:4503): protocol: write tcp 188.166.225.170:52988->104.36.82.21:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:23.962443430 (58h46m33.68500651s) [noise=true , address=104.36.82.43:4838 , piece_id=7IJV4BGZJQEOC2ZAQH5FASVFALOBSMHRUYHMJXZF6UKFX52ZZ4MQ, seg_pos={0 0}, upload=56390]: Done uploading piece. err: ecclient: upload failed (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6, address:104.36.82.43:4838): protocol: write tcp 188.166.225.170:44168->104.36.82.43:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:23.962465382 (58h46m33.685028452s) [seg_pos={0 0}, upload=56390]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:22:24.202833359 (58h46m33.925396399s) [upload=56390]: Done uploading segments... Jul 26 22:22:24.202860260 (58h46m33.92542331s) [upload=56390]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.242035034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X...............] duration: 5.983778299s FAIL (2025-07-26 22:22:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.98380518s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 18.002406529s FAIL (2025-07-26 22:22:15 UTC) orbiter warning: operation error S3: GetObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1855ED318630ED12, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: unable to create order limits) Duration: 18.002427349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X..X...........X....................] duration: 5.977704306s FAIL (2025-07-26 22:22:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.977728844s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X..............] duration: 5.83515246s FAIL (2025-07-26 22:21:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.835171657s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X...............] duration: 5.809262808s FAIL (2025-07-26 22:21:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.809288335s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..X...........X...................] duration: 5.803824263s FAIL (2025-07-26 22:21:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.803893296s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.227867981s FAIL (2025-07-26 22:21:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 072760 (58h44m34.3996358s) [noise=true , address=104.36.82.44:4903 , piece_id=LQ677J25Q635VROJA6JCUJ33QCQMM5WDSYXCN362CJRRFBXH2NUQ, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.677447858 (58h44m34.400010898s) [noise=true , address=104.36.82.40:4968 , piece_id=ENKG5X3CZJNVCGC26PMWBYGZ3LWWIFVZY2V46A2Z3TDSUV5RY5QQ, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.677847873 (58h44m34.400410903s) [noise=true , address=77.90.31.16:4752 , piece_id=XPANYBESY3DSTUETDOO2NRTM4NPXVXHQDFUNZNORDFMM2XE3YKIA, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.678092101 (58h44m34.400655141s) [noise=true , address=45.91.116.5:4310 , piece_id=LLB6LWUN6E3HO2T75PIBXPBWFQKUD4VKGVFXJJ5A72N2KFIH7I4A, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.678378678 (58h44m34.400941719s) [noise=true , address=64.49.13.138:4483 , piece_id=FITKRJ4673SIHJCPGRKY7BYR5W2UYMBK266N6K45OBYD7EKBKQ5A, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.678591817 (58h44m34.401154857s) [noise=true , address=104.36.82.41:4773 , piece_id=AW7S2BN4I33MXE5NFHHF4JUDBYGF5IL2UH7UYSZ7MWDWF26Q5EGQ, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.678977354 (58h44m34.401540434s) [noise=true , address=104.36.82.39:4736 , piece_id=ZYS4JQAGCMGXVFXR3SM2GYOB3IFRDWXHZGRFUGHGBMHTIQERUTGA, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.679394512 (58h44m34.401957562s) [noise=true , address=156.146.43.225:4449 , piece_id=GVVZAQLG6TITIHCLYS6TFVCVCQ6HHI5YNVRHG76ZDFZZ3LA6VCLQ, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.679687593 (58h44m34.402250633s) [noise=true , address=104.36.82.42:4810 , piece_id=55256S2GMLJYVFVRN2GSNVJVY4EB3PDQDZ6KHNRABD2XHPZ26O3Q, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.680032614 (58h44m34.402595654s) [noise=true , address=104.36.82.19:4416 , piece_id=SURTCJOYQ33UAF7Z73Y4IMILYRDFABQBFF7ZU5RVCOQSG5C2HN3Q, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.680419143 (58h44m34.402982183s) [noise=true , address=104.36.82.34:4574 , piece_id=Q2AL7EGFDVMO6FZF6BIOHKFENWNTGN6N3O23IEXW6SFLZSPIMGPA, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:24.680660003 (58h44m34.403223053s) [noise=true , address=79.127.201.215:4345 , piece_id=GTAURRSSWGV33L74DIYBRSAVCVLX7JAP6K2N5I5QE7JKSRERYLZA, seg_pos={0 0}, upload=56375]: Uploading piece... Jul 26 22:20:25.695717879 (58h44m35.418280919s) [noise=true , address=156.146.43.225:4449 , piece_id=GVVZAQLG6TITIHCLYS6TFVCVCQ6HHI5YNVRHG76ZDFZZ3LA6VCLQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:20:25.820676839 (58h44m35.543239879s) [upload=56375]: Next returned nil segment Jul 26 22:20:25.820691377 (58h44m35.543254417s) [upload=56375]: Waiting for error group managing segments... Jul 26 22:20:26.241113957 (58h44m35.963676997s) [noise=true , address=193.149.164.143:4702 , piece_id=RATDQNOP3YMXK5ACD3XJCXJ4ZFYV2AHGMDLJ4NIO3GBNW44YDJEQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: <nil> Jul 26 22:20:27.832241957 (58h44m37.554804997s) [noise=true , address=64.49.13.138:4483 , piece_id=FITKRJ4673SIHJCPGRKY7BYR5W2UYMBK266N6K45OBYD7EKBKQ5A, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:20:30.278226357 (58h44m40.000789387s) [noise=true , address=45.91.116.13:4175 , piece_id=DOYE364WBOAFBXP2MCHZLAG7ZTGQR67TJCQLSM3PZPR22ZHH3CRQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:20:30.449225434 (58h44m40.171788474s) [noise=true , address=79.127.219.38:4004 , piece_id=DN3H7KDKCMPFE322SBLWIXSTQWRDO5V5SKCBBHO4ZZGCVXL46HOA, seg_pos={0 0}, upload=56375]: Done uploading piece. err: <nil> Jul 26 22:20:42.115523618 (58h44m51.838086698s) [noise=true , address=45.91.116.5:4310 , piece_id=LLB6LWUN6E3HO2T75PIBXPBWFQKUD4VKGVFXJJ5A72N2KFIH7I4A, seg_pos={0 0}, upload=56375]: Done uploading piece. err: <nil> Jul 26 22:20:42.884472964 (58h44m52.607036004s) [noise=true , address=79.127.201.215:4345 , piece_id=GTAURRSSWGV33L74DIYBRSAVCVLX7JAP6K2N5I5QE7JKSRERYLZA, seg_pos={0 0}, upload=56375]: Done uploading piece. err: <nil> Jul 26 22:20:57.033178247 (58h45m6.755741287s) [noise=true , address=77.90.31.16:4752 , piece_id=XPANYBESY3DSTUETDOO2NRTM4NPXVXHQDFUNZNORDFMM2XE3YKIA, seg_pos={0 0}, upload=56375]: Done uploading piece. err: <nil> Jul 26 22:21:23.505031892 (58h45m33.227594932s) [noise=true , address=104.36.82.34:4574 , piece_id=Q2AL7EGFDVMO6FZF6BIOHKFENWNTGN6N3O23IEXW6SFLZSPIMGPA, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:12ZnB3zQPaoZ54HYpp4XayHrxiZSsC76wj6wk426PPu6tWFspMV, address:104.36.82.34:4574): protocol: write tcp 188.166.225.170:57090->104.36.82.34:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505076919 (58h45m33.227639949s) [noise=true , address=104.36.82.21:4508 , piece_id=ZUES4ZL577TZGOYPOR25ZZ7LXFNC7RIWEUX3VUPEXO5WOT5TC6PQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:104.36.82.21:4508): protocol: rpc: dial tcp 104.36.82.21:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505120773 (58h45m33.227683813s) [noise=true , address=104.36.82.17:4334 , piece_id=NFNDH7YZ23DHKFV3EZSBSQSO74WDCOD7DOX2O276ISWKMF75ABQQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1, address:104.36.82.17:4334): protocol: write tcp 188.166.225.170:42982->104.36.82.17:4334: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505152724 (58h45m33.227715764s) [noise=true , address=104.36.82.39:4736 , piece_id=ZYS4JQAGCMGXVFXR3SM2GYOB3IFRDWXHZGRFUGHGBMHTIQERUTGA, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:126MWCnfveMZSncssS6wf4HGCi4eXvL7FvMD31oYoiR3SEFdGxX, address:104.36.82.39:4736): protocol: write tcp 188.166.225.170:47774->104.36.82.39:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505162532 (58h45m33.227725572s) [noise=true , address=104.36.82.44:4903 , piece_id=LQ677J25Q635VROJA6JCUJ33QCQMM5WDSYXCN362CJRRFBXH2NUQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: write tcp 188.166.225.170:53886->104.36.82.44:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505178423 (58h45m33.227741463s) [noise=true , address=104.36.82.19:4416 , piece_id=SURTCJOYQ33UAF7Z73Y4IMILYRDFABQBFF7ZU5RVCOQSG5C2HN3Q, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:12Rwbh1TjuvXvNEcTKsS49FMV3YKJUUzea9K8zFdDXns2ZyMcvg, address:104.36.82.19:4416): protocol: write tcp 188.166.225.170:42780->104.36.82.19:4416: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505209853 (58h45m33.227772883s) [noise=true , address=104.36.82.41:4773 , piece_id=AW7S2BN4I33MXE5NFHHF4JUDBYGF5IL2UH7UYSZ7MWDWF26Q5EGQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV, address:104.36.82.41:4773): protocol: write tcp 188.166.225.170:49314->104.36.82.41:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505215834 (58h45m33.227778874s) [noise=true , address=104.36.82.38:4690 , piece_id=KX5FA437L6TGXDBHXFMYA5LPPRBBTLT4D2W7SBN2ELF365H6CNNQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:104.36.82.38:4690): protocol: rpc: dial tcp 104.36.82.38:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505219591 (58h45m33.227782621s) [noise=true , address=104.36.82.40:4968 , piece_id=ENKG5X3CZJNVCGC26PMWBYGZ3LWWIFVZY2V46A2Z3TDSUV5RY5QQ, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:1J98jLqn1SVMYjTYxKnNqYbRxSQ5vaUnq2G7XpVngHTaWRJ8oX, address:104.36.82.40:4968): protocol: write tcp 188.166.225.170:34546->104.36.82.40:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505236163 (58h45m33.227799204s) [noise=true , address=104.36.82.42:4810 , piece_id=55256S2GMLJYVFVRN2GSNVJVY4EB3PDQDZ6KHNRABD2XHPZ26O3Q, seg_pos={0 0}, upload=56375]: Done uploading piece. err: ecclient: upload failed (node:1CGnNCsXi1RAnTx9P28561Q1wK6LBqpBzqyuowxaAZJETQRz2w, address:104.36.82.42:4810): protocol: write tcp 188.166.225.170:52748->104.36.82.42:4810: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:23.505256032 (58h45m33.227819062s) [seg_pos={0 0}, upload=56375]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:21:23.730974872 (58h45m33.453537942s) [upload=56375]: Done uploading segments... Jul 26 22:21:23.731006762 (58h45m33.453569802s) [upload=56375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.227886826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 15.428519273s FAIL (2025-07-26 22:21:16 UTC) orbiter warning: operation error S3: GetObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1855ED318630ED12, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: unable to create order limits) Duration: 15.428764461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.............] duration: 5.723342202s FAIL (2025-07-26 22:21:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.723369896s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X..............] duration: 6.954250001s FAIL (2025-07-26 22:21:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.954282915s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..X...........X..................] duration: 5.940472265s FAIL (2025-07-26 22:20:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.940502103s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X............] duration: 5.636309791s FAIL (2025-07-26 22:20:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.636337414s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.............] duration: 5.732135081s FAIL (2025-07-26 22:20:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.732197811s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 32.769279018s FAIL (2025-07-26 22:20:27 UTC) orbiter warning: operation error S3: GetObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1855ED318630ED12, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: unable to create order limits) Duration: 32.769290119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.229883995s FAIL (2025-07-26 22:20:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== {0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.182830537 (58h43m33.905393577s) [noise=true , address=77.90.31.13:4644 , piece_id=T42FP4GSHNTF5AR5SM7Y62SRFQJPL76YSXZXTZSDBXVXNC7QD2FA, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.182955907 (58h43m33.905518947s) [noise=true , address=79.127.205.245:4223 , piece_id=4DKY2BA4YAFCIB7H5WGMYLC5SSUSW4DOL73YSD4MSGW3342TH6IA, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.183115703 (58h43m33.905678733s) [noise=true , address=104.36.82.39:4745 , piece_id=5PD33SRKLYLRFKEZLBKOH3WR42GJYEP3CX7ACB6MJYD7MVTHENMQ, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.183307580 (58h43m33.90587062s) [noise=true , address=104.36.82.55:4195 , piece_id=IQCOVGXI2F44MRX5DKN2TZQM6SIILF5OZB4U2TJJD6RJOJAV4R3A, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.183431637 (58h43m33.905994677s) [noise=true , address=104.36.82.47:4534 , piece_id=GM2VGAGXNR46Z7SR65VDLOWYZXUX7KJ7ILEX3HJPSXTFRJXJXZ2Q, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.183714538 (58h43m33.906277578s) [noise=true , address=104.36.82.20:4450 , piece_id=V5IQH6OB6MHDBMECDJCDMPWB23EYARBKNOGQT4LBAQ5PWYU46XDQ, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.183841391 (58h43m33.906404431s) [noise=true , address=104.36.82.44:4889 , piece_id=FH75BRRD54IVDL5CJ7EIKSVXOGOJN7WGD6U3GXGK3CKYNLT4RJ7A, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.184050091 (58h43m33.906613131s) [noise=true , address=79.127.201.215:4360 , piece_id=DXHW422RG4OXY4WLXZON7J5EMIHRLYSH7L5AF3ZH7TQPCTX45QNQ, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.184136105 (58h43m33.906699135s) [noise=true , address=104.36.82.46:4506 , piece_id=3OZBHNZILFVBUQFLCORXLDPF622TPKPDUCR4OIHXXRC3SNGF4EHQ, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.184388698 (58h43m33.906951738s) [noise=true , address=104.36.82.52:4094 , piece_id=5KSEWCLBHNRVTAF24QA6ZCWEJBWED2VHQ7NKFBQYF26F3DHDBEPA, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.184462971 (58h43m33.907026011s) [noise=true , address=79.127.219.38:4005 , piece_id=273BLQW6VEKIJKV3AYCNRTDNZQ23WVFIU2G2MBLQIUQU2HXHRMTQ, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.185100540 (58h43m33.90766357s) [noise=true , address=104.36.82.2:4749 , piece_id=47XSTGAUBITPT6RGVTCVAM4I3SOQJCFKT5SVDKKNNKDLKUZQDELQ, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:24.185279883 (58h43m33.907842923s) [noise=true , address=104.36.82.22:4516 , piece_id=VF47BPDMJATHUYMWDBSPD4CJNVOPOBZ3VMFTHWGYGZYAVXZ34RPQ, seg_pos={0 0}, upload=56355]: Uploading piece... Jul 26 22:19:25.318489713 (58h43m35.041052753s) [upload=56355]: Next returned nil segment Jul 26 22:19:25.318508359 (58h43m35.041071399s) [upload=56355]: Waiting for error group managing segments... Jul 26 22:19:25.400487677 (58h43m35.123050718s) [noise=true , address=64.49.13.138:4482 , piece_id=CA62AR6CWMRR35UDYUSN2NYOOS7VYSLSRTDWJZECHX4QYHCFFF3Q, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:19:25.631216863 (58h43m35.353779903s) [noise=true , address=79.127.205.245:4223 , piece_id=4DKY2BA4YAFCIB7H5WGMYLC5SSUSW4DOL73YSD4MSGW3342TH6IA, seg_pos={0 0}, upload=56355]: Done uploading piece. err: <nil> Jul 26 22:19:25.632122305 (58h43m35.354685345s) [noise=true , address=79.127.205.235:4679 , piece_id=OOIEK26FSW3KMSUVXJ26VY5GL3USG73IOQISBYVKH37R66TEJNBA, seg_pos={0 0}, upload=56355]: Done uploading piece. err: <nil> Jul 26 22:19:25.636817088 (58h43m35.359380118s) [noise=true , address=79.127.205.234:4605 , piece_id=32UZGLUBOVJCEVSMIIFFFTMKTEOMONC55YXMHJIEO3ADP342JRRA, seg_pos={0 0}, upload=56355]: Done uploading piece. err: <nil> Jul 26 22:19:26.160442452 (58h43m35.883005492s) [noise=true , address=79.127.219.38:4005 , piece_id=273BLQW6VEKIJKV3AYCNRTDNZQ23WVFIU2G2MBLQIUQU2HXHRMTQ, seg_pos={0 0}, upload=56355]: Done uploading piece. err: <nil> Jul 26 22:19:27.963787119 (58h43m37.686350159s) [noise=true , address=77.90.31.13:4644 , piece_id=T42FP4GSHNTF5AR5SM7Y62SRFQJPL76YSXZXTZSDBXVXNC7QD2FA, seg_pos={0 0}, upload=56355]: Done uploading piece. err: <nil> Jul 26 22:19:30.686431069 (58h43m40.408994099s) [noise=true , address=79.127.201.215:4360 , piece_id=DXHW422RG4OXY4WLXZON7J5EMIHRLYSH7L5AF3ZH7TQPCTX45QNQ, seg_pos={0 0}, upload=56355]: Done uploading piece. err: <nil> Jul 26 22:20:23.043500292 (58h44m32.766063332s) [noise=true , address=104.36.82.53:4146 , piece_id=LR42N57KCHYJYV3KLWAUZLWY6HC2FTC6ZT4OPAKSSD3A34WKMKRQ, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 188.166.225.170:50258->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.043547723 (58h44m32.766110763s) [noise=true , address=104.36.82.8:4974 , piece_id=KGIYERJONUHSXXRTJR4NJIN2ZAYPXDCJZOJFKWRFM3SSWEYY3Q3Q, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1, address:104.36.82.8:4974): protocol: rpc: dial tcp 104.36.82.8:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.043577199 (58h44m32.766140239s) [noise=true , address=104.36.82.47:4534 , piece_id=GM2VGAGXNR46Z7SR65VDLOWYZXUX7KJ7ILEX3HJPSXTFRJXJXZ2Q, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: rpc: dial tcp 104.36.82.47:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.043968337 (58h44m32.766531377s) [noise=true , address=104.36.82.2:4749 , piece_id=47XSTGAUBITPT6RGVTCVAM4I3SOQJCFKT5SVDKKNNKDLKUZQDELQ, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:12sRotj46WjNNsVciJDRvSEbrj8XHwshKRmMneedu6UH4egmZDm, address:104.36.82.2:4749): protocol: rpc: dial tcp 104.36.82.2:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.044004416 (58h44m32.766567456s) [noise=true , address=104.36.82.39:4745 , piece_id=5PD33SRKLYLRFKEZLBKOH3WR42GJYEP3CX7ACB6MJYD7MVTHENMQ, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:1ujPZUMvwigdNuqXMzvgaxXdiVQzEkfsSDNQY7BujAnHLHkXks, address:104.36.82.39:4745): protocol: rpc: dial tcp 104.36.82.39:4745: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.044031257 (58h44m32.766594298s) [noise=true , address=104.36.82.55:4195 , piece_id=IQCOVGXI2F44MRX5DKN2TZQM6SIILF5OZB4U2TJJD6RJOJAV4R3A, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: rpc: dial tcp 104.36.82.55:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.044061525 (58h44m32.766624565s) [noise=true , address=104.36.82.52:4094 , piece_id=5KSEWCLBHNRVTAF24QA6ZCWEJBWED2VHQ7NKFBQYF26F3DHDBEPA, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 188.166.225.170:34716->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.044248193 (58h44m32.766811223s) [noise=true , address=104.36.82.22:4516 , piece_id=VF47BPDMJATHUYMWDBSPD4CJNVOPOBZ3VMFTHWGYGZYAVXZ34RPQ, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:104.36.82.22:4516): protocol: write tcp 188.166.225.170:49008->104.36.82.22:4516: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.044284483 (58h44m32.766847513s) [noise=true , address=104.36.82.20:4450 , piece_id=V5IQH6OB6MHDBMECDJCDMPWB23EYARBKNOGQT4LBAQ5PWYU46XDQ, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:12UeL1LySKQySa8wjvG4FBs69cE1iWkW4Diz83nu1PxBCKxyvki, address:104.36.82.20:4450): protocol: rpc: dial tcp 104.36.82.20:4450: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.044307897 (58h44m32.766870937s) [noise=true , address=104.36.82.46:4506 , piece_id=3OZBHNZILFVBUQFLCORXLDPF622TPKPDUCR4OIHXXRC3SNGF4EHQ, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: write tcp 188.166.225.170:44704->104.36.82.46:4506: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.044535824 (58h44m32.767098874s) [noise=true , address=104.36.82.44:4889 , piece_id=FH75BRRD54IVDL5CJ7EIKSVXOGOJN7WGD6U3GXGK3CKYNLT4RJ7A, seg_pos={0 0}, upload=56355]: Done uploading piece. err: ecclient: upload failed (node:1PCtJRFHPHi1dHJNcRnGZg8HZPQDfo5L3aw6DgVnUzVUz92Kyv, address:104.36.82.44:4889): protocol: rpc: dial tcp 104.36.82.44:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:23.044574026 (58h44m32.767137066s) [seg_pos={0 0}, upload=56355]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:20:23.271785773 (58h44m32.994348813s) [upload=56355]: Done uploading segments... Jul 26 22:20:23.271813406 (58h44m32.994376446s) [upload=56355]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.229898503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X............] duration: 5.75186604s FAIL (2025-07-26 22:20:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.75189173s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X...........] duration: 5.713859172s FAIL (2025-07-26 22:19:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.713882777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X...........] duration: 5.831289588s FAIL (2025-07-26 22:19:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.831319845s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..X...........X.................] duration: 6.32574992s FAIL (2025-07-26 22:19:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.325773896s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.229018731s FAIL (2025-07-26 22:19:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== _id=K2JASMLMM64N56Z6GIBVWUH7QRRGMK7Q3O4LVKNQ5VN5NH4PZVKA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.678925147 (58h42m33.401488187s) [noise=true , address=79.127.219.38:4004 , piece_id=HGZBV6OMUZXQ5537YLVNSNNROCE72C6UGX6H3OVL7YXMOMNDQ36A, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.679267461 (58h42m33.401830501s) [noise=true , address=104.36.82.13:4204 , piece_id=ZTSC5GRH7BF5QC3D7SBNNVGCLCTHLIOXM2WYT2GWR5TDMXW3VSLQ, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.679544863 (58h42m33.402107903s) [noise=true , address=104.36.82.22:4528 , piece_id=I7ZPBCPV2JXI6S5XKXYGQDELORRHICC5G5JSDKAFRAMPL3VGOKIA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.679610518 (58h42m33.402173548s) [noise=true , address=104.36.82.16:4313 , piece_id=XOOU5SOA5DQJ6D3MOZKTPRFMR77BBIGTYGWLI2OMCXW6JJ4JE7VA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.679898519 (58h42m33.402461579s) [noise=true , address=79.127.201.215:4356 , piece_id=RHLFHZQJ2Z5RJSZIYXOIDILXFYVYDTFBNE46MPOW3VAGOPIQWX5Q, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680091658 (58h42m33.402654698s) [noise=true , address=79.127.205.245:4244 , piece_id=MALMEUEXLITFZLFR6GSO6IJNY5H7XIZMMHWOOG4RCFBHOXHGRG2Q, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680224172 (58h42m33.402787232s) [noise=true , address=79.127.205.234:4619 , piece_id=4XDWA465NYPBZG3CRAR3MPQ325AI6UMGN7NIVDQ35PNAP5S4NT7Q, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680541359 (58h42m33.403104409s) [noise=true , address=104.36.82.39:4748 , piece_id=H5HV5PPSWTNFERDGC4YTCVLIOL2HXSLHFLPOS4WT64TERYRTFL6A, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680567159 (58h42m33.403130219s) [noise=true , address=104.36.82.5:4869 , piece_id=Y4KYWSS67PHD4UASJFDQJV7GNFP6X3AOKQVVNOEZ23BIV5SY767Q, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680613346 (58h42m33.403176386s) [noise=true , address=64.49.13.138:4483 , piece_id=3T233IH7NYOMC5YOZ654WHJ3BFNQEXIVJQHLM5M6JD2PRRNCSGCA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680879546 (58h42m33.403442596s) [noise=true , address=79.127.205.246:4275 , piece_id=TGUMN64TQVEY62SRYG3WML65TPZO6OS7GTUER4JQ7LSUNDTXS3DA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.680947156 (58h42m33.403510196s) [noise=true , address=104.36.82.53:4124 , piece_id=KJRXOP4RMHH5RAH6P3LWSFJVFZU6OYCEI3V5445FGJ3SN2UUX2KA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681113714 (58h42m33.403676755s) [noise=true , address=104.36.82.46:4488 , piece_id=TIHWYNS3GVJCKBOOK3JFJIT7L37255ASMHWTGFJCEKATL36FQ2RQ, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681348673 (58h42m33.403911714s) [noise=true , address=104.36.82.47:4536 , piece_id=PLAHWXGDAXPOLPELRK7GJZCSSGFGRJJPVKLYJ7T27QLW6Y73ZSDQ, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681380484 (58h42m33.403943524s) [noise=true , address=77.90.31.17:4774 , piece_id=FYW7VEOMUSGWF7I3IPSCPP7SOSN3VEBO7HCHKXU7HGLYYFUHA6KA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681411303 (58h42m33.403974333s) [noise=true , address=79.127.205.233:4647 , piece_id=7TDQDE5V63EFA4XBI4F2J4CIXU3NZHZN6PQSPIOIF2HTFYF6DRBQ, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:23.681708111 (58h42m33.404271151s) [noise=true , address=77.90.31.16:4747 , piece_id=4BKBJM3EBPYF3CZGQBNGOMZIJQ42DPLYLE4GFXORWFG6XQIIBFUA, seg_pos={0 0}, upload=56339]: Uploading piece... Jul 26 22:18:24.768905801 (58h42m34.491468831s) [upload=56339]: Next returned nil segment Jul 26 22:18:24.768924407 (58h42m34.491487447s) [upload=56339]: Waiting for error group managing segments... Jul 26 22:18:25.078517169 (58h42m34.801080199s) [noise=true , address=79.127.205.245:4244 , piece_id=MALMEUEXLITFZLFR6GSO6IJNY5H7XIZMMHWOOG4RCFBHOXHGRG2Q, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.088054276 (58h42m34.810617316s) [noise=true , address=79.127.205.234:4619 , piece_id=4XDWA465NYPBZG3CRAR3MPQ325AI6UMGN7NIVDQ35PNAP5S4NT7Q, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.206373843 (58h42m34.928936873s) [noise=true , address=64.49.13.138:4483 , piece_id=3T233IH7NYOMC5YOZ654WHJ3BFNQEXIVJQHLM5M6JD2PRRNCSGCA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.420195831 (58h42m35.142758871s) [noise=true , address=79.127.205.246:4275 , piece_id=TGUMN64TQVEY62SRYG3WML65TPZO6OS7GTUER4JQ7LSUNDTXS3DA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.674385704 (58h42m35.396948744s) [noise=true , address=77.90.31.16:4747 , piece_id=4BKBJM3EBPYF3CZGQBNGOMZIJQ42DPLYLE4GFXORWFG6XQIIBFUA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:25.744652173 (58h42m35.467215213s) [noise=true , address=79.127.205.233:4647 , piece_id=7TDQDE5V63EFA4XBI4F2J4CIXU3NZHZN6PQSPIOIF2HTFYF6DRBQ, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:26.136907322 (58h42m35.859470352s) [noise=true , address=79.127.219.38:4004 , piece_id=HGZBV6OMUZXQ5537YLVNSNNROCE72C6UGX6H3OVL7YXMOMNDQ36A, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:37.804852967 (58h42m47.527416007s) [noise=true , address=79.127.201.215:4356 , piece_id=RHLFHZQJ2Z5RJSZIYXOIDILXFYVYDTFBNE46MPOW3VAGOPIQWX5Q, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:18:40.903611577 (58h42m50.626174607s) [noise=true , address=77.90.31.17:4774 , piece_id=FYW7VEOMUSGWF7I3IPSCPP7SOSN3VEBO7HCHKXU7HGLYYFUHA6KA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: <nil> Jul 26 22:19:22.591611584 (58h43m32.314174624s) [noise=true , address=104.36.82.22:4528 , piece_id=I7ZPBCPV2JXI6S5XKXYGQDELORRHICC5G5JSDKAFRAMPL3VGOKIA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1nDzCSeAbqSKpZUVGcu4NEj32jsoqojqSpzfpKYoXciXEjsTwo, address:104.36.82.22:4528): protocol: write tcp 188.166.225.170:48014->104.36.82.22:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:22.591784053 (58h43m32.314347104s) [noise=true , address=104.36.82.56:4256 , piece_id=K2JASMLMM64N56Z6GIBVWUH7QRRGMK7Q3O4LVKNQ5VN5NH4PZVKA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: write tcp 188.166.225.170:34668->104.36.82.56:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:22.591832116 (58h43m32.314395156s) [noise=true , address=104.36.82.13:4204 , piece_id=ZTSC5GRH7BF5QC3D7SBNNVGCLCTHLIOXM2WYT2GWR5TDMXW3VSLQ, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: write tcp 188.166.225.170:46316->104.36.82.13:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:22.591887001 (58h43m32.314450031s) [noise=true , address=104.36.82.46:4488 , piece_id=TIHWYNS3GVJCKBOOK3JFJIT7L37255ASMHWTGFJCEKATL36FQ2RQ, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: read tcp 188.166.225.170:45498->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:22.591989417 (58h43m32.314552457s) [noise=true , address=104.36.82.5:4869 , piece_id=Y4KYWSS67PHD4UASJFDQJV7GNFP6X3AOKQVVNOEZ23BIV5SY767Q, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:104.36.82.5:4869): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592065652 (58h43m32.314628692s) [noise=true , address=104.36.82.47:4536 , piece_id=PLAHWXGDAXPOLPELRK7GJZCSSGFGRJJPVKLYJ7T27QLW6Y73ZSDQ, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592191143 (58h43m32.314754183s) [noise=true , address=104.36.82.53:4124 , piece_id=KJRXOP4RMHH5RAH6P3LWSFJVFZU6OYCEI3V5445FGJ3SN2UUX2KA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592287257 (58h43m32.314850297s) [noise=true , address=104.36.82.39:4748 , piece_id=H5HV5PPSWTNFERDGC4YTCVLIOL2HXSLHFLPOS4WT64TERYRTFL6A, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1aZ4HiQxxV3qYJLYViMi7me9mRnPPBgPihW8fwzeMTKXnwbUb9, address:104.36.82.39:4748): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592567743 (58h43m32.315130783s) [noise=true , address=104.36.82.16:4313 , piece_id=XOOU5SOA5DQJ6D3MOZKTPRFMR77BBIGTYGWLI2OMCXW6JJ4JE7VA, seg_pos={0 0}, upload=56339]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:104.36.82.16:4313): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:22.592587370 (58h43m32.31515042s) [seg_pos={0 0}, upload=56339]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:19:22.818960500 (58h43m32.54152354s) [upload=56339]: Done uploading segments... Jul 26 22:19:22.818986520 (58h43m32.54154956s) [upload=56339]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.229035542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X..........] duration: 6.047730745s FAIL (2025-07-26 22:19:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.047760101s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [....XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X..........] duration: 5.821891719s FAIL (2025-07-26 22:18:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.821978265s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X..X...........X................] duration: 5.976140637s FAIL (2025-07-26 22:18:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.976160817s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [...XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.........] duration: 6.504105325s FAIL (2025-07-26 22:18:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.504133508s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232130842s FAIL (2025-07-26 22:18:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 66EIXDB6SRXFZS62UN3DDFEG6VGE2L7QQMSSIDBEGDKRZYIF2SA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196026007 (58h41m32.918589047s) [noise=true , address=104.36.82.47:4523 , piece_id=6WEFNRREMKK2DDL3AJ6EOBDUL65NVRG5QPP67TPSNCQFG7Q7SGTQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196245748 (58h41m32.918808788s) [noise=true , address=104.36.82.46:4504 , piece_id=56AG7BBZYIRTPAZXBBXIXMNDMVDWIV7FZZLUUDEWPANYCVIPRWAQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196536875 (58h41m32.919099915s) [noise=true , address=79.127.219.48:4090 , piece_id=TU5ROLJUOEVUGQK7KRXQHUVZBOTRPPIV4ZRH4DAHPBJDET4JSMNA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196749150 (58h41m32.91931219s) [noise=true , address=79.127.205.245:4239 , piece_id=36ZMWZZSE7SUWSXLTGVPETZPF47R3IT4MG3CEREZYRUOCWUNHKJQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.196975875 (58h41m32.919538925s) [noise=true , address=104.36.82.13:4199 , piece_id=AE7BEEVNGRXCBYQNDOAQWJDU4J6664F2RXUQFZAWBFZXU77ABWUQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.197158333 (58h41m32.919721373s) [noise=true , address=104.36.82.53:4126 , piece_id=RQWX4PKHX5O6CMPISMGBSAD2FCQMZUIE2NLAQH7PZJC3TOJM3TYA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.197382081 (58h41m32.919945111s) [noise=true , address=104.36.82.56:4230 , piece_id=JREE7B6OBC4MJQAGLC6Y72ADSC5BPKWBPUNDLMA56OJ4DGOSLQTQ, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.197635847 (58h41m32.920198887s) [noise=true , address=104.36.82.19:4403 , piece_id=6HHV56YAPJ4SYDI6FLYCT7QXSOB7LK3RW2QSRYCLU5AYBVCVXNLA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.197862281 (58h41m32.920425311s) [noise=true , address=104.36.82.52:4120 , piece_id=X6U4HJC4VNOQN5UDNHQQLWHUWTDVQ6CFXSOXH7Z2JNYYPM7XWZSA, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:23.198047524 (58h41m32.920610564s) [noise=true , address=104.36.82.16:4321 , piece_id=XFRVYB3YKGZXMNQLD36HAOLKIPFZBJFKGFRGZ4XNMAH75QETHE3A, seg_pos={0 0}, upload=56324]: Uploading piece... Jul 26 22:17:24.306114851 (58h41m34.028677891s) [upload=56324]: Next returned nil segment Jul 26 22:17:24.306135130 (58h41m34.02869817s) [upload=56324]: Waiting for error group managing segments... Jul 26 22:17:24.619782982 (58h41m34.342346022s) [noise=true , address=79.127.205.245:4239 , piece_id=36ZMWZZSE7SUWSXLTGVPETZPF47R3IT4MG3CEREZYRUOCWUNHKJQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:17:25.112253227 (58h41m34.834816267s) [noise=true , address=193.149.164.143:4696 , piece_id=SD276I3U4LO5UGN7ATRGEY3J6PEYKCQ53TMHXKGKJIEK3HW5BYLQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:17:28.157631716 (58h41m37.880194756s) [noise=true , address=156.146.43.225:4434 , piece_id=G66EIXDB6SRXFZS62UN3DDFEG6VGE2L7QQMSSIDBEGDKRZYIF2SA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:17:28.365760502 (58h41m38.088323542s) [noise=true , address=79.127.219.48:4090 , piece_id=TU5ROLJUOEVUGQK7KRXQHUVZBOTRPPIV4ZRH4DAHPBJDET4JSMNA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:17:29.925769153 (58h41m39.648332193s) [noise=true , address=79.127.201.215:4343 , piece_id=PIZKLF7GFJNT4XD4V5CXTVRRZX67LQ3MRVQCAGAT77EWAYEMDMGA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: <nil> Jul 26 22:18:22.104689925 (58h42m31.827252965s) [noise=true , address=104.36.82.53:4126 , piece_id=RQWX4PKHX5O6CMPISMGBSAD2FCQMZUIE2NLAQH7PZJC3TOJM3TYA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: write tcp 188.166.225.170:57708->104.36.82.53:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104743468 (58h42m31.827306508s) [noise=true , address=104.36.82.47:4523 , piece_id=6WEFNRREMKK2DDL3AJ6EOBDUL65NVRG5QPP67TPSNCQFG7Q7SGTQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:18:22.104759909 (58h42m31.827322939s) [noise=true , address=104.36.82.14:4255 , piece_id=HLIJHJKIWLZDD2P5SFZTTNATHCHKGDF6ROOV6KNEVGF5F7VDBZAA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:104.36.82.14:4255): protocol: write tcp 188.166.225.170:41178->104.36.82.14:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104801308 (58h42m31.827364348s) [noise=true , address=104.36.82.46:4504 , piece_id=56AG7BBZYIRTPAZXBBXIXMNDMVDWIV7FZZLUUDEWPANYCVIPRWAQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:38374->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104834241 (58h42m31.827397281s) [noise=true , address=104.36.82.56:4230 , piece_id=JREE7B6OBC4MJQAGLC6Y72ADSC5BPKWBPUNDLMA56OJ4DGOSLQTQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: write tcp 188.166.225.170:40952->104.36.82.56:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104868937 (58h42m31.827431977s) [noise=true , address=104.36.82.19:4403 , piece_id=6HHV56YAPJ4SYDI6FLYCT7QXSOB7LK3RW2QSRYCLU5AYBVCVXNLA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: write tcp 188.166.225.170:53990->104.36.82.19:4403: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104897903 (58h42m31.827460943s) [noise=true , address=104.36.82.9:4029 , piece_id=BGV32JJN2EJWGK5ENQU5ZVVO5QJLFNYC56PLE562KK5EJKZLC65A, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12snsP1y5fbxY8XrDoXihzQrsBxJ5Y38C2jSGhHtV1ukN426eiP, address:104.36.82.9:4029): protocol: write tcp 188.166.225.170:52076->104.36.82.9:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104932109 (58h42m31.827495149s) [noise=true , address=104.36.82.16:4321 , piece_id=XFRVYB3YKGZXMNQLD36HAOLKIPFZBJFKGFRGZ4XNMAH75QETHE3A, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae, address:104.36.82.16:4321): protocol: write tcp 188.166.225.170:49368->104.36.82.16:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.104951756 (58h42m31.827514796s) [noise=true , address=104.36.82.13:4199 , piece_id=AE7BEEVNGRXCBYQNDOAQWJDU4J6664F2RXUQFZAWBFZXU77ABWUQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7, address:104.36.82.13:4199): protocol: write tcp 188.166.225.170:35932->104.36.82.13:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105003204 (58h42m31.827566244s) [noise=true , address=104.36.82.36:4618 , piece_id=FHDVYJTTGNWIOIIKTWW2W73NJGXUMKPNVB2XPXQVC5HQCW6N6MFQ, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:104.36.82.36:4618): protocol: write tcp 188.166.225.170:50378->104.36.82.36:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105049964 (58h42m31.827613004s) [noise=true , address=104.36.82.55:4199 , piece_id=NE36CQPKCGYLLUXVP34OWYM4D4JZ3NJYZ4VTZVJGUDCCT4J6BZEA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: write tcp 188.166.225.170:45378->104.36.82.55:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105055615 (58h42m31.827618655s) [noise=true , address=104.36.82.49:4006 , piece_id=ZTAGNKD74XZIMSVDVVCJH27NR5PAH3C56JGKOVMZZBFNHUW3W43A, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:12oMAD2S6soQVtMpXxakmXz6DobyVZk9fQ3Lp4UxczMdQj5eHGW, address:104.36.82.49:4006): protocol: write tcp 188.166.225.170:53964->104.36.82.49:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105081294 (58h42m31.827644334s) [noise=true , address=104.36.82.52:4120 , piece_id=X6U4HJC4VNOQN5UDNHQQLWHUWTDVQ6CFXSOXH7Z2JNYYPM7XWZSA, seg_pos={0 0}, upload=56324]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 188.166.225.170:33994->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:22.105118164 (58h42m31.827681204s) [seg_pos={0 0}, upload=56324]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:18:22.335351420 (58h42m32.05791446s) [upload=56324]: Done uploading segments... Jul 26 22:18:22.335368874 (58h42m32.057931914s) [upload=56324]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.232161891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.....XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.........] duration: 5.73263894s FAIL (2025-07-26 22:18:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.732670941s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X..X...........X...............] duration: 6.268740663s FAIL (2025-07-26 22:17:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.268760852s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [....XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X........] duration: 5.821600469s FAIL (2025-07-26 22:17:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.821624794s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X..X...........X..............] duration: 6.090158441s FAIL (2025-07-26 22:17:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.090179561s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X........] duration: 6.469878973s FAIL (2025-07-26 22:17:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.469901315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235372645s FAIL (2025-07-26 22:17:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== oise=true , address=64.49.13.138:4482 , piece_id=IIQUNJVK2ZKQM6E6SE62TJQP6AH2ALZWPTUFTOBQBIA46ITUH53Q, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726111329 (58h40m32.44867437s) [noise=true , address=104.36.82.56:4240 , piece_id=KPTRUPKVA2HICFT255KCGL7HPVXIJALAXAATHJ2U2LRRUN6MRI5Q, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726250676 (58h40m32.448813746s) [noise=true , address=104.36.82.9:4038 , piece_id=LOLIDLMYRWUFIAIWAFMGUFBINTZIYCQWDRMT7ZUM4PMIS27HFUKA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726381086 (58h40m32.448944126s) [noise=true , address=104.36.82.4:4815 , piece_id=HSYQ2TB7KQZBZBISESPDM5BDZZHZ336W52JCSJKVP5YRG2WFOVKA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726551522 (58h40m32.449114552s) [noise=true , address=104.36.82.47:4544 , piece_id=QJ5AKJAA6R2WUJZLCLVVDXTTXJKGXZIW7D72OMVWCC65MARZHZFA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726661161 (58h40m32.449224242s) [noise=true , address=193.149.164.143:4696 , piece_id=Z4SSXTYM3CG32CGQYEW5WK3RCZJMJ7MYYBCIB2GYSAJWZTYOBAYA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726835625 (58h40m32.449398665s) [noise=true , address=104.36.82.52:4110 , piece_id=7UGDQ6UOWC2CZREUXWOSSZW77IFFHIPVN6BIRYGXKQPOTF7BS2WA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.726894247 (58h40m32.449457277s) [noise=true , address=104.36.82.14:4226 , piece_id=FZ4TYOEP7WTSBCIYCRTKENBZMWJOSY223TTEI5M4PCAA7HY7MEOQ, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.727079872 (58h40m32.449642912s) [noise=true , address=77.90.31.10:4525 , piece_id=JY2LGV4RKE26JRPBY5NDWUN7TQA2WZXG4QZOTJ2UVRTWO6JI26GQ, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.727098709 (58h40m32.449661749s) [noise=true , address=45.91.116.14:4222 , piece_id=7TXNNKDTNZYVWGJSRHYVPMOSLPUZZAJ62F5QX7LO73WR2NXGJFNQ, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.727300995 (58h40m32.449864035s) [noise=true , address=79.127.205.246:4283 , piece_id=MVGCJDBV7TRYOAJXXYG3MMHZFQLOUEW4VC67TGS7Q7IZY43BEFRA, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:22.727516488 (58h40m32.450079518s) [noise=true , address=104.36.82.5:4864 , piece_id=ZPP26JUJFUBPEBL4QQ42WIT4PJNWPUFUTG3A4AXQUGRRZXNF3M5Q, seg_pos={0 0}, upload=56309]: Uploading piece... Jul 26 22:16:23.832997277 (58h40m33.555560307s) [upload=56309]: Next returned nil segment Jul 26 22:16:23.833018188 (58h40m33.555581228s) [upload=56309]: Waiting for error group managing segments... Jul 26 22:16:24.148692978 (58h40m33.871256008s) [noise=true , address=79.127.205.246:4283 , piece_id=MVGCJDBV7TRYOAJXXYG3MMHZFQLOUEW4VC67TGS7Q7IZY43BEFRA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:24.244664336 (58h40m33.967227396s) [noise=true , address=64.49.13.138:4482 , piece_id=IIQUNJVK2ZKQM6E6SE62TJQP6AH2ALZWPTUFTOBQBIA46ITUH53Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:24.682786220 (58h40m34.40534926s) [noise=true , address=45.91.116.14:4222 , piece_id=7TXNNKDTNZYVWGJSRHYVPMOSLPUZZAJ62F5QX7LO73WR2NXGJFNQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:24.783583341 (58h40m34.506146381s) [noise=true , address=77.90.31.10:4525 , piece_id=JY2LGV4RKE26JRPBY5NDWUN7TQA2WZXG4QZOTJ2UVRTWO6JI26GQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:24.848011319 (58h40m34.570574359s) [noise=true , address=45.91.116.10:4100 , piece_id=A5IDMUPIVPRLS2OQE54WYOEMAF2BFDTGELGJJX4WNYX5AM4ZZJHQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:16:25.508948087 (58h40m35.231511127s) [noise=true , address=193.149.164.143:4696 , piece_id=Z4SSXTYM3CG32CGQYEW5WK3RCZJMJ7MYYBCIB2GYSAJWZTYOBAYA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: <nil> Jul 26 22:17:21.629292218 (58h41m31.351855248s) [noise=true , address=104.36.82.9:4038 , piece_id=LOLIDLMYRWUFIAIWAFMGUFBINTZIYCQWDRMT7ZUM4PMIS27HFUKA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12VTVM9YuevPA4vsWExjpyW5B9pqyEKKruUwnj1jNhE3j1ScCBg, address:104.36.82.9:4038): protocol: rpc: dial tcp 104.36.82.9:4038: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629336443 (58h41m31.351899483s) [noise=true , address=104.36.82.46:4501 , piece_id=ENPOEH4NO55CY7GKGWTKNZZM3IZEVYZ4HSITZCJOPWJE6S3NGS5Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 188.166.225.170:59472->104.36.82.46:4501: use of closed network connection; write tcp 188.166.225.170:59472->104.36.82.46:4501: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59472->104.36.82.46:4501: use of closed network connection Jul 26 22:17:21.629373313 (58h41m31.351936354s) [noise=true , address=104.36.82.52:4110 , piece_id=7UGDQ6UOWC2CZREUXWOSSZW77IFFHIPVN6BIRYGXKQPOTF7BS2WA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629390537 (58h41m31.351953567s) [noise=true , address=104.36.82.56:4240 , piece_id=KPTRUPKVA2HICFT255KCGL7HPVXIJALAXAATHJ2U2LRRUN6MRI5Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:129QqkFEzZdtzQ6HAKxPWxDNgDVUNJJLH59oRVXnHExrwkDECyf, address:104.36.82.56:4240): protocol: rpc: dial tcp 104.36.82.56:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629412589 (58h41m31.351975619s) [noise=true , address=104.36.82.14:4226 , piece_id=FZ4TYOEP7WTSBCIYCRTKENBZMWJOSY223TTEI5M4PCAA7HY7MEOQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:104.36.82.14:4226): protocol: rpc: dial tcp 104.36.82.14:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629426696 (58h41m31.351989736s) [noise=true , address=104.36.82.5:4864 , piece_id=ZPP26JUJFUBPEBL4QQ42WIT4PJNWPUFUTG3A4AXQUGRRZXNF3M5Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629457484 (58h41m31.352020525s) [noise=true , address=104.36.82.55:4200 , piece_id=KFAJFMI4S22TZAWTLZ3HLRSG2AO7D6XLHQR2ZYC6JM5OC3WPJIRQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: write tcp 188.166.225.170:57996->104.36.82.55:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629461783 (58h41m31.352024823s) [noise=true , address=104.36.82.50:4033 , piece_id=ZMWTGQALLTZU5CQO6SZEZBUXURM7IBXMFJQZOPQCEDWJSMEDV67Q, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1GhnCxqMdrDsaY3SrRs7VmHbSn8htrbQ3zJy2B5Nb77awMkCR6, address:104.36.82.50:4033): protocol: rpc: dial tcp 104.36.82.50:4033: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629498674 (58h41m31.352061704s) [noise=true , address=104.36.82.22:4533 , piece_id=ZZCVN53UFYRFPZ5Q6OZQUGLJE4S3PHXU4JEDYOA4LMQVCUJ4LYWQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12LXK5pbXavbPZh4YSuEkgk7wsRqwvcVEHv4w9FDRYqSkEU89VF, address:104.36.82.22:4533): protocol: rpc: dial tcp 104.36.82.22:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629505406 (58h41m31.352068446s) [noise=true , address=104.36.82.4:4815 , piece_id=HSYQ2TB7KQZBZBISESPDM5BDZZHZ336W52JCSJKVP5YRG2WFOVKA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:104.36.82.4:4815): protocol: rpc: dial tcp 104.36.82.4:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629512760 (58h41m31.3520758s) [noise=true , address=104.36.82.47:4544 , piece_id=QJ5AKJAA6R2WUJZLCLVVDXTTXJKGXZIW7D72OMVWCC65MARZHZFA, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:104.36.82.47:4544): protocol: write tcp 188.166.225.170:40436->104.36.82.47:4544: use of closed network connection; write tcp 188.166.225.170:40436->104.36.82.47:4544: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40436->104.36.82.47:4544: use of closed network connection Jul 26 22:17:21.629536876 (58h41m31.352099917s) [noise=true , address=104.36.82.19:4432 , piece_id=L4AGQGOZC2CGULIEIGBCSFDVRYFOG7CJVHLUJOABHSRVD6AL4CBQ, seg_pos={0 0}, upload=56309]: Done uploading piece. err: ecclient: upload failed (node:12oh9XZaiLhaZdMyq1cx88yrrG9C9sHtH3RTA6W2Ly39t23ArSQ, address:104.36.82.19:4432): protocol: rpc: dial tcp 104.36.82.19:4432: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:21.629550764 (58h41m31.352113804s) [seg_pos={0 0}, upload=56309]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:17:21.862172550 (58h41m31.5847356s) [upload=56309]: Done uploading segments... Jul 26 22:17:21.862202046 (58h41m31.584765136s) [upload=56309]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.235398415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.079885847s FAIL (2025-07-26 22:17:10 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 2.079916325s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..X...........X.............] duration: 6.117555573s FAIL (2025-07-26 22:16:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.11757451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.999162382s FAIL (2025-07-26 22:16:50 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.999205874s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.......] duration: 25.537360956s FAIL (2025-07-26 22:16:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 25.537388087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.......] duration: 6.092028138s FAIL (2025-07-26 22:16:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.092058788s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235598673s FAIL (2025-07-26 22:16:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 19.51:4212 , piece_id=ZOCGFN2RIKHLSVJ3TP7XDYKVNFXUJDRBIGODBHGCE6KIMHWKJRGA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.632586689 (58h39m28.355149719s) [noise=true , address=104.36.82.56:4241 , piece_id=VPGRZQN5EN7U5AYKOSJ32WJV3T5DN7JLROH2JVPTSZGO7FAFKNKA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.632966185 (58h39m28.355529225s) [noise=true , address=104.36.82.52:4103 , piece_id=ORKV4CDOSGLMHJ6PUNDA57HQBXE5C4YB27IGWJOGVMFDJQILJSKA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.633074873 (58h39m28.355637903s) [noise=true , address=79.127.205.246:4284 , piece_id=TORDDF7JXLTMX2XTEZBK33FAEZNWTO2UAXTASAG5YLGJ2UV46TFA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.633481221 (58h39m28.356044261s) [noise=true , address=104.36.82.45:4937 , piece_id=7CNFHQG7I56N42H5IPA2UXOP3CRFM6E4HGYDDFSZQNEIGHDR4H4A, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.633666655 (58h39m28.356229695s) [noise=true , address=104.36.82.47:4539 , piece_id=4B5CTYR56TG63BWAGRI7ONMEQVJPFBWICVWI4QE54I6SIJUAUTUA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.634074675 (58h39m28.356637715s) [noise=true , address=79.127.219.48:4090 , piece_id=CZJJV6VFNQKFBY6O6BZF5IVN6EZJJTIZRWD6PB6YJ44X4LLJRZAA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.634096898 (58h39m28.356659938s) [noise=true , address=79.127.219.38:4007 , piece_id=G7UG364ZLUWWJGRSD2KB34XR57JRPYPPUI5YN57HQDLZTWUEXFEQ, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.634508395 (58h39m28.357071436s) [noise=true , address=104.36.82.18:4370 , piece_id=AZUNHNW5C3F7KZTHXZWKG63XPSMSRWBHO6RWUMFHV7TT6EWTJEDA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.634614759 (58h39m28.357177789s) [noise=true , address=104.36.82.9:4022 , piece_id=LOQ743ADRSZKAZZTG566EZUCFXY4XN6ZOLR5FIDHRQD6ICCNYVKA, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.635239104 (58h39m28.357802144s) [noise=true , address=77.90.31.13:4644 , piece_id=KSZEKKW6LKFV5BXAY2EPMU7WQ2K5IEYAJKUFNA4OVSWCF75XFC5Q, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.635460307 (58h39m28.358023347s) [noise=true , address=156.146.43.225:4434 , piece_id=APB2HOBJMCZJBLOPJKL2SGEVC5X6YA2ZGI3SOFKZDC32T6XYRRLQ, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:18.635657063 (58h39m28.358220104s) [noise=true , address=104.36.82.6:4903 , piece_id=B7RI6UHZCTBQRML42OXXL6FGPWD6CDVM2PJH2YTHVK6HWQ5SWNXQ, seg_pos={0 0}, upload=56288]: Uploading piece... Jul 26 22:15:19.785734719 (58h39m29.508297759s) [upload=56288]: Next returned nil segment Jul 26 22:15:19.785751402 (58h39m29.508314442s) [upload=56288]: Waiting for error group managing segments... Jul 26 22:15:20.077111762 (58h39m29.799674802s) [noise=true , address=79.127.205.246:4284 , piece_id=TORDDF7JXLTMX2XTEZBK33FAEZNWTO2UAXTASAG5YLGJ2UV46TFA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:20.632258635 (58h39m30.354821675s) [noise=true , address=77.90.31.13:4644 , piece_id=KSZEKKW6LKFV5BXAY2EPMU7WQ2K5IEYAJKUFNA4OVSWCF75XFC5Q, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:20.877974995 (58h39m30.600538035s) [noise=true , address=79.127.219.38:4007 , piece_id=G7UG364ZLUWWJGRSD2KB34XR57JRPYPPUI5YN57HQDLZTWUEXFEQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:21.185022439 (58h39m30.907585479s) [noise=true , address=79.127.219.48:4090 , piece_id=CZJJV6VFNQKFBY6O6BZF5IVN6EZJJTIZRWD6PB6YJ44X4LLJRZAA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:24.671453279 (58h39m34.394016319s) [noise=true , address=79.127.219.51:4212 , piece_id=ZOCGFN2RIKHLSVJ3TP7XDYKVNFXUJDRBIGODBHGCE6KIMHWKJRGA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:25.874699680 (58h39m35.59726272s) [noise=true , address=156.146.43.225:4434 , piece_id=APB2HOBJMCZJBLOPJKL2SGEVC5X6YA2ZGI3SOFKZDC32T6XYRRLQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:15:26.710314392 (58h39m36.432877432s) [noise=true , address=79.127.201.215:4351 , piece_id=EJC6HH266U4KVMIXOG5DJHWPSDL4BEIAAPGNPJADLYY3HI4RB7UQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: <nil> Jul 26 22:16:17.545961274 (58h40m27.268524314s) [noise=true , address=104.36.82.9:4022 , piece_id=LOQ743ADRSZKAZZTG566EZUCFXY4XN6ZOLR5FIDHRQD6ICCNYVKA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: write tcp 188.166.225.170:54868->104.36.82.9:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546011050 (58h40m27.26857409s) [noise=true , address=104.36.82.53:4128 , piece_id=IVQWI4UQ7UYP3NBP4AFCTA7VLLJCUJX2ZYQVYOSXZFQZEWFKEKVQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 188.166.225.170:38354->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546034475 (58h40m27.268597515s) [noise=true , address=104.36.82.34:4552 , piece_id=TIZKHAFVGSKWE422F66JI6KBEYPJZMUEDWRZ7C3JTO2ZJBAPUWHA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:12YmWSZZQgXm2gdZCahnyCdSzxYPAerzCfD38mWPmpTDyGYT823, address:104.36.82.34:4552): protocol: write tcp 188.166.225.170:58424->104.36.82.34:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546100711 (58h40m27.268663751s) [noise=true , address=104.36.82.46:4483 , piece_id=HANC4VCEXEXBKNTYX7FJF6N3KT2FOREABTK72AV2THJZUS2WWHFQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: write tcp 188.166.225.170:52922->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546150797 (58h40m27.268713837s) [noise=true , address=104.36.82.47:4539 , piece_id=4B5CTYR56TG63BWAGRI7ONMEQVJPFBWICVWI4QE54I6SIJUAUTUA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: write tcp 188.166.225.170:48568->104.36.82.47:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546184010 (58h40m27.26874705s) [noise=true , address=104.36.82.56:4241 , piece_id=VPGRZQN5EN7U5AYKOSJ32WJV3T5DN7JLROH2JVPTSZGO7FAFKNKA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: write tcp 188.166.225.170:45984->104.36.82.56:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546317967 (58h40m27.268880997s) [noise=true , address=104.36.82.45:4937 , piece_id=7CNFHQG7I56N42H5IPA2UXOP3CRFM6E4HGYDDFSZQNEIGHDR4H4A, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94, address:104.36.82.45:4937): protocol: write tcp 188.166.225.170:47492->104.36.82.45:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546354637 (58h40m27.268917677s) [noise=true , address=104.36.82.18:4370 , piece_id=AZUNHNW5C3F7KZTHXZWKG63XPSMSRWBHO6RWUMFHV7TT6EWTJEDA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm, address:104.36.82.18:4370): protocol: write tcp 188.166.225.170:34922->104.36.82.18:4370: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546372291 (58h40m27.268935331s) [noise=true , address=104.36.82.52:4103 , piece_id=ORKV4CDOSGLMHJ6PUNDA57HQBXE5C4YB27IGWJOGVMFDJQILJSKA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 188.166.225.170:53508->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546394033 (58h40m27.268957073s) [noise=true , address=104.36.82.55:4212 , piece_id=GJ7T2EMA7IECPRT4GUPGB6U43T73MBIJEMP4XZGFF4VMYWECB6VA, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: write tcp 188.166.225.170:60236->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546399072 (58h40m27.268962112s) [noise=true , address=104.36.82.6:4903 , piece_id=B7RI6UHZCTBQRML42OXXL6FGPWD6CDVM2PJH2YTHVK6HWQ5SWNXQ, seg_pos={0 0}, upload=56288]: Done uploading piece. err: ecclient: upload failed (node:1zFBuDaxAtbFho2wumoXWSoQbB7y9VaSFizz31gV8n1fLK6syE, address:104.36.82.6:4903): protocol: write tcp 188.166.225.170:52932->104.36.82.6:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:17.546418780 (58h40m27.26898182s) [seg_pos={0 0}, upload=56288]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:16:17.779435262 (58h40m27.501998302s) [upload=56288]: Done uploading segments... Jul 26 22:16:17.779494345 (58h40m27.502057385s) [upload=56288]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.235612078s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)