Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X...X......X.XX...XXXX.....X..........X..X..X...XX.......] duration: 5.860691618s FAIL (2025-07-27 00:41:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.860713569s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.820388197s FAIL (2025-07-27 00:41:19 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.820413424s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [X.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..] duration: 6.038375126s FAIL (2025-07-27 00:41:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== LR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:56638->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867442766 (61h3m39.590005806s) [seg_pos={0 0}, upload=58544]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:39:30.101162500 (61h3m39.82372554s) [upload=58544]: Done uploading segments... Jul 27 00:39:30.101188930 (61h3m39.82375197s) [upload=58544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:39:41.811972819 (61h3m51.534535859s) [upload=58553]: Starting upload Jul 27 00:39:41.812045497 (61h3m51.534608537s) [upload=58553]: Done starting upload Jul 27 00:39:41.812141992 (61h3m51.534705032s) [upload=58553]: Uploading segments... Jul 27 00:39:41.812278373 (61h3m51.534841413s) [upload=58553]: Got next segment. Inline: false Jul 27 00:39:41.812310535 (61h3m51.534873575s) [upload=58553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:39:41.812315604 (61h3m51.534878644s) [upload=58553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:39:42.051587579 (61h3m51.774150619s) [seg_pos={0 0}, upload=58553]: Begin upload segment... Jul 27 00:39:42.051634690 (61h3m51.77419773s) [seg_pos={0 0}, upload=58553]: Done begin upload segment. Jul 27 00:39:42.051651631 (61h3m51.774214672s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.051658735 (61h3m51.774221775s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052079060 (61h3m51.7746421s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052419801 (61h3m51.774982841s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052674629 (61h3m51.775237669s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052943233 (61h3m51.775506273s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053183753 (61h3m51.775746793s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053425346 (61h3m51.775988386s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053692406 (61h3m51.776255446s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053907798 (61h3m51.776470838s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054094305 (61h3m51.776657345s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054274550 (61h3m51.776837581s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054601535 (61h3m51.777164565s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054800016 (61h3m51.777363046s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054989448 (61h3m51.777552488s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.055169392 (61h3m51.777732432s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:43.801433736 (61h3m53.523996776s) [upload=58553]: Next returned nil segment Jul 27 00:39:43.801450518 (61h3m53.524013558s) [upload=58553]: Waiting for error group managing segments... Jul 27 00:39:44.079598064 (61h3m53.802161104s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.158666595 (61h3m53.881229625s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.166883606 (61h3m53.889446646s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174165408 (61h3m53.896728448s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174342528 (61h3m53.896905568s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.180598436 (61h3m53.903161476s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.219950789 (61h3m53.942513829s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.232932967 (61h3m53.955496007s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.323809349 (61h3m54.04637239s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:45.501672536 (61h3m55.224235576s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:46.178232591 (61h3m55.900795631s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:40:11.165533280 (61h4m20.88809632s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165602982 (61h4m20.888166022s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165659470 (61h4m20.88822251s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165747348 (61h4m20.888310388s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165843924 (61h4m20.888406964s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:33372->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165864824 (61h4m20.888427864s) [seg_pos={0 0}, upload=58553]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:40:11.394295096 (61h4m21.116858186s) [upload=58553]: Done uploading segments... Jul 27 00:40:11.394328439 (61h4m21.116891519s) [upload=58553]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.038401729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.247202589s FAIL (2025-07-27 00:41:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 205.245:4224 , piece_id=UWWRGFFTGW745YAIWJPNWENQH7ODDNTLKV3T6MYIW7MGYLFET2QQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.778788856 (61h4m13.501351896s) [noise=true , address=45.91.116.8:4062 , piece_id=EZHA3GLP75N3K3USKVJM2DEA7SXDMXHVFGULJZO2JTUO4ZI36RIQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779031020 (61h4m13.50159407s) [noise=true , address=77.90.31.11:4574 , piece_id=WK62EN2C5XUPBOXRDYJMYSR3VIODNWCBLCBATNCEWFSP54CPNXUQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779093169 (61h4m13.501656209s) [noise=true , address=79.127.201.215:4359 , piece_id=TGE2EKKVJHHI2NYBUOREPEKAY2JLLYKYOXKTZKMIZ2KDAV3MF47A, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779266820 (61h4m13.501829871s) [noise=true , address=77.90.31.21:4909 , piece_id=W3JH3ATUSBJW7JP3S5HKWFZPD2P753J3F2RSQR6LMSRCZECJ7TFA, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779346723 (61h4m13.501909784s) [noise=true , address=104.36.82.46:4499 , piece_id=VSCRLBZI4PP7C44BIVDEL5AQ4WQXPQJLZN5YCVGWD4OK7S2U6PQA, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779592634 (61h4m13.502155684s) [noise=true , address=45.91.116.11:4130 , piece_id=OQCVPHLFS5TKB7LGNKAF55FMBEOUK5MO2NH5Q2M24HGD45I33JQQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779599256 (61h4m13.502162296s) [noise=true , address=79.127.219.38:4006 , piece_id=R6QVCRQA2HXCT3FJLAQUHDJ6JP3RLM3Q3NJYSK6OK5D54E6TZNCQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.779912216 (61h4m13.502475276s) [noise=true , address=104.36.82.54:4177 , piece_id=JJDHXHGGORLE4D5S6GUTCVTKZJAVG72UQTEHUMVJFKJ6Z5DYAUVQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780018949 (61h4m13.502581989s) [noise=true , address=104.36.82.52:4085 , piece_id=S7LKZA4FYQZ5OF3GB2M3NMVCJGFDK47WFXMZ2G2WTV6ZHTJK3X7Q, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780189997 (61h4m13.502753037s) [noise=true , address=79.127.205.235:4688 , piece_id=IJPC7JNYBQDVL73WXLJDUTHHGPP64VLXA5NRHEBDPFBJ263PP6MQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780258408 (61h4m13.502821458s) [noise=true , address=79.127.205.234:4606 , piece_id=LZLYSPYE6G53UGKIJGV3CTMADCESKKAEVAB5AHKZRKXDXRH7IBEQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780439975 (61h4m13.503003015s) [noise=true , address=104.36.82.42:4823 , piece_id=XWJH5AOQKPHF6562VXXFD577IDTHGCBAKNFKVQDX6IXUQKMBFVYA, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.780510309 (61h4m13.503073359s) [noise=true , address=45.91.116.4:4964 , piece_id=MA3SA3FDE2TYN7NPTNYJP5ACFDPRQRTVTNFEP5AYV45F2YC7LG7A, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.781423437 (61h4m13.503986477s) [noise=true , address=104.36.82.47:4524 , piece_id=NUQXFYU5S6L6MUJ75MRLWY5B3QQOPKPEMFFSE4X65APN3HFSCYFQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:03.781667844 (61h4m13.504230894s) [noise=true , address=77.90.31.17:4784 , piece_id=UZMGJM6EXE63H5MVFGUKV2KDQ5UENNXBL7YVXBOSVP7VDZ4QK3YQ, seg_pos={0 0}, upload=58558]: Uploading piece... Jul 27 00:40:04.880942902 (61h4m14.603505942s) [upload=58558]: Next returned nil segment Jul 27 00:40:04.880958422 (61h4m14.603521462s) [upload=58558]: Waiting for error group managing segments... Jul 27 00:40:05.172884399 (61h4m14.895447439s) [noise=true , address=79.127.205.234:4606 , piece_id=LZLYSPYE6G53UGKIJGV3CTMADCESKKAEVAB5AHKZRKXDXRH7IBEQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:05.326184366 (61h4m15.048747406s) [noise=true , address=79.127.205.245:4224 , piece_id=UWWRGFFTGW745YAIWJPNWENQH7ODDNTLKV3T6MYIW7MGYLFET2QQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:05.502534094 (61h4m15.225097144s) [noise=true , address=79.127.205.235:4688 , piece_id=IJPC7JNYBQDVL73WXLJDUTHHGPP64VLXA5NRHEBDPFBJ263PP6MQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:05.750199488 (61h4m15.472762528s) [noise=true , address=45.91.116.8:4062 , piece_id=EZHA3GLP75N3K3USKVJM2DEA7SXDMXHVFGULJZO2JTUO4ZI36RIQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:05.764593127 (61h4m15.487156167s) [noise=true , address=79.127.219.38:4006 , piece_id=R6QVCRQA2HXCT3FJLAQUHDJ6JP3RLM3Q3NJYSK6OK5D54E6TZNCQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:06.077637180 (61h4m15.80020023s) [noise=true , address=77.90.31.11:4574 , piece_id=WK62EN2C5XUPBOXRDYJMYSR3VIODNWCBLCBATNCEWFSP54CPNXUQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:10.015760555 (61h4m19.738323595s) [noise=true , address=45.91.116.4:4964 , piece_id=MA3SA3FDE2TYN7NPTNYJP5ACFDPRQRTVTNFEP5AYV45F2YC7LG7A, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:15.262299954 (61h4m24.984862984s) [noise=true , address=45.91.116.9:4277 , piece_id=LI6H7STY7PC6KKWC346EB6342YR56FSBELYTGY4ZL4Y7IS5GVYCA, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:40:17.833459686 (61h4m27.556022726s) [noise=true , address=45.91.116.11:4130 , piece_id=OQCVPHLFS5TKB7LGNKAF55FMBEOUK5MO2NH5Q2M24HGD45I33JQQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:40:37.513430104 (61h4m47.235993144s) [noise=true , address=79.127.201.215:4359 , piece_id=TGE2EKKVJHHI2NYBUOREPEKAY2JLLYKYOXKTZKMIZ2KDAV3MF47A, seg_pos={0 0}, upload=58558]: Done uploading piece. err: <nil> Jul 27 00:41:02.672822384 (61h5m12.395385424s) [noise=true , address=104.36.82.52:4085 , piece_id=S7LKZA4FYQZ5OF3GB2M3NMVCJGFDK47WFXMZ2G2WTV6ZHTJK3X7Q, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:35886->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672847302 (61h5m12.395410342s) [noise=true , address=104.36.82.46:4499 , piece_id=VSCRLBZI4PP7C44BIVDEL5AQ4WQXPQJLZN5YCVGWD4OK7S2U6PQA, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 188.166.225.170:39188->104.36.82.46:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672880225 (61h5m12.395443265s) [noise=true , address=104.36.82.47:4524 , piece_id=NUQXFYU5S6L6MUJ75MRLWY5B3QQOPKPEMFFSE4X65APN3HFSCYFQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: rpc: dial tcp 104.36.82.47:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672902898 (61h5m12.395465938s) [noise=true , address=104.36.82.42:4823 , piece_id=XWJH5AOQKPHF6562VXXFD577IDTHGCBAKNFKVQDX6IXUQKMBFVYA, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: rpc: dial tcp 104.36.82.42:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672924129 (61h5m12.395487169s) [noise=true , address=104.36.82.54:4177 , piece_id=JJDHXHGGORLE4D5S6GUTCVTKZJAVG72UQTEHUMVJFKJ6Z5DYAUVQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 188.166.225.170:42474->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672957903 (61h5m12.395520943s) [noise=true , address=104.36.82.14:4233 , piece_id=73MXEVAOSLY2GXT4PIPH2EPRZLLAFZYLN3ZMEBZQQMXUWQKPIPHQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:124wXnmRuxo3o7AVwdSbUmai3SebaMV5qgScGVFSWFKSRvBNTyB, address:104.36.82.14:4233): protocol: write tcp 188.166.225.170:57380->104.36.82.14:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.672961260 (61h5m12.3955243s) [noise=true , address=77.90.31.17:4784 , piece_id=UZMGJM6EXE63H5MVFGUKV2KDQ5UENNXBL7YVXBOSVP7VDZ4QK3YQ, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: write tcp 188.166.225.170:49658->77.90.31.17:4784: use of closed network connection; write tcp 188.166.225.170:49658->77.90.31.17:4784: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49658->77.90.31.17:4784: use of closed network connection Jul 27 00:41:02.672997288 (61h5m12.395560328s) [noise=true , address=77.90.31.21:4909 , piece_id=W3JH3ATUSBJW7JP3S5HKWFZPD2P753J3F2RSQR6LMSRCZECJ7TFA, seg_pos={0 0}, upload=58558]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 188.166.225.170:52332->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.673020443 (61h5m12.395583483s) [seg_pos={0 0}, upload=58558]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:41:02.918063519 (61h5m12.640626559s) [upload=58558]: Done uploading segments... Jul 27 00:41:02.918091382 (61h5m12.640654422s) [upload=58558]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.247215343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX......] duration: 7.61103776s FAIL (2025-07-27 00:40:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.611245738s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX.] duration: 7.318737466s FAIL (2025-07-27 00:40:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== LR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:56638->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867442766 (61h3m39.590005806s) [seg_pos={0 0}, upload=58544]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:39:30.101162500 (61h3m39.82372554s) [upload=58544]: Done uploading segments... Jul 27 00:39:30.101188930 (61h3m39.82375197s) [upload=58544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:39:41.811972819 (61h3m51.534535859s) [upload=58553]: Starting upload Jul 27 00:39:41.812045497 (61h3m51.534608537s) [upload=58553]: Done starting upload Jul 27 00:39:41.812141992 (61h3m51.534705032s) [upload=58553]: Uploading segments... Jul 27 00:39:41.812278373 (61h3m51.534841413s) [upload=58553]: Got next segment. Inline: false Jul 27 00:39:41.812310535 (61h3m51.534873575s) [upload=58553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:39:41.812315604 (61h3m51.534878644s) [upload=58553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:39:42.051587579 (61h3m51.774150619s) [seg_pos={0 0}, upload=58553]: Begin upload segment... Jul 27 00:39:42.051634690 (61h3m51.77419773s) [seg_pos={0 0}, upload=58553]: Done begin upload segment. Jul 27 00:39:42.051651631 (61h3m51.774214672s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.051658735 (61h3m51.774221775s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052079060 (61h3m51.7746421s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052419801 (61h3m51.774982841s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052674629 (61h3m51.775237669s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052943233 (61h3m51.775506273s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053183753 (61h3m51.775746793s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053425346 (61h3m51.775988386s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053692406 (61h3m51.776255446s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053907798 (61h3m51.776470838s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054094305 (61h3m51.776657345s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054274550 (61h3m51.776837581s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054601535 (61h3m51.777164565s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054800016 (61h3m51.777363046s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054989448 (61h3m51.777552488s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.055169392 (61h3m51.777732432s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:43.801433736 (61h3m53.523996776s) [upload=58553]: Next returned nil segment Jul 27 00:39:43.801450518 (61h3m53.524013558s) [upload=58553]: Waiting for error group managing segments... Jul 27 00:39:44.079598064 (61h3m53.802161104s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.158666595 (61h3m53.881229625s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.166883606 (61h3m53.889446646s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174165408 (61h3m53.896728448s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174342528 (61h3m53.896905568s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.180598436 (61h3m53.903161476s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.219950789 (61h3m53.942513829s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.232932967 (61h3m53.955496007s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.323809349 (61h3m54.04637239s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:45.501672536 (61h3m55.224235576s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:46.178232591 (61h3m55.900795631s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:40:11.165533280 (61h4m20.88809632s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165602982 (61h4m20.888166022s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165659470 (61h4m20.88822251s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165747348 (61h4m20.888310388s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165843924 (61h4m20.888406964s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:33372->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165864824 (61h4m20.888427864s) [seg_pos={0 0}, upload=58553]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:40:11.394295096 (61h4m21.116858186s) [upload=58553]: Done uploading segments... Jul 27 00:40:11.394328439 (61h4m21.116891519s) [upload=58553]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.318771231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [XX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..X] duration: 30.233844013s FAIL (2025-07-27 00:40:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.233898618s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X....] duration: 32.695061306s FAIL (2025-07-27 00:40:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.695080031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.830264335s FAIL (2025-07-27 00:40:33 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.830285996s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX] duration: 30.231884295s FAIL (2025-07-27 00:40:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== LR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:56638->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867442766 (61h3m39.590005806s) [seg_pos={0 0}, upload=58544]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:39:30.101162500 (61h3m39.82372554s) [upload=58544]: Done uploading segments... Jul 27 00:39:30.101188930 (61h3m39.82375197s) [upload=58544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:39:41.811972819 (61h3m51.534535859s) [upload=58553]: Starting upload Jul 27 00:39:41.812045497 (61h3m51.534608537s) [upload=58553]: Done starting upload Jul 27 00:39:41.812141992 (61h3m51.534705032s) [upload=58553]: Uploading segments... Jul 27 00:39:41.812278373 (61h3m51.534841413s) [upload=58553]: Got next segment. Inline: false Jul 27 00:39:41.812310535 (61h3m51.534873575s) [upload=58553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:39:41.812315604 (61h3m51.534878644s) [upload=58553]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:39:42.051587579 (61h3m51.774150619s) [seg_pos={0 0}, upload=58553]: Begin upload segment... Jul 27 00:39:42.051634690 (61h3m51.77419773s) [seg_pos={0 0}, upload=58553]: Done begin upload segment. Jul 27 00:39:42.051651631 (61h3m51.774214672s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.051658735 (61h3m51.774221775s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052079060 (61h3m51.7746421s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052419801 (61h3m51.774982841s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052674629 (61h3m51.775237669s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.052943233 (61h3m51.775506273s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053183753 (61h3m51.775746793s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053425346 (61h3m51.775988386s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053692406 (61h3m51.776255446s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.053907798 (61h3m51.776470838s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054094305 (61h3m51.776657345s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054274550 (61h3m51.776837581s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054601535 (61h3m51.777164565s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054800016 (61h3m51.777363046s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.054989448 (61h3m51.777552488s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:42.055169392 (61h3m51.777732432s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Uploading piece... Jul 27 00:39:43.801433736 (61h3m53.523996776s) [upload=58553]: Next returned nil segment Jul 27 00:39:43.801450518 (61h3m53.524013558s) [upload=58553]: Waiting for error group managing segments... Jul 27 00:39:44.079598064 (61h3m53.802161104s) [noise=true , address=45.91.116.8:4048 , piece_id=QUYNO4TH2BLGNZNYSTPDB2FPBTXS4CKWB2WNLBLKMHRCWBKMVFNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.158666595 (61h3m53.881229625s) [noise=true , address=45.91.116.11:4117 , piece_id=LYEQXKGMEF3L2UNUBBRCCHGPTGAYRGKP5SF3LT2GGN6XYTF5IYAQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.166883606 (61h3m53.889446646s) [noise=true , address=45.91.116.9:4262 , piece_id=AD2XYKOMG3QGSMAHES5NJ7RV6CLQUBDEY3QOB5CBVM7YDO3WRKXA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174165408 (61h3m53.896728448s) [noise=true , address=77.90.31.18:4807 , piece_id=7ALX7T3LLGXR6UZNGMFQUR262VA6DQF5VHUEWPHG7AX5D4NYALNA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.174342528 (61h3m53.896905568s) [noise=true , address=45.91.116.4:4940 , piece_id=E6M5YH6UFVZOWBFDFUBYRGZE3P6SHRZLUSG4Y6ENKBNF5SXOKXSA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.180598436 (61h3m53.903161476s) [noise=true , address=45.91.116.14:4191 , piece_id=TGJ5NXWKEHORD5XA2VOZX42ZB7FAYQIAJ7IIN6JACBNNEVFVLNJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.219950789 (61h3m53.942513829s) [noise=true , address=77.90.31.14:4662 , piece_id=HHZJFZLJGKIVDDLKTVAJ75VCJUI2M3BBCALHD3O7O4T2PM62PYVQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.232932967 (61h3m53.955496007s) [noise=true , address=45.91.116.13:4155 , piece_id=UJ2BX6YSIVL3T4A5EL6MATFJLFZGQAQDDEJZHW6QHSWGEI7CRXTQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:44.323809349 (61h3m54.04637239s) [noise=true , address=45.91.116.6:4973 , piece_id=23T6EZUKCVFR7PDAVJJO35GPZ63PRYJX6XM7XSWJ63QW4IVX2DJA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:45.501672536 (61h3m55.224235576s) [noise=true , address=77.90.31.13:4628 , piece_id=ZIVV2N5H47OOH7UXN7SO5NDVJK5TAV2SJLBTS2IEG4NZOWZWFNNQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:39:46.178232591 (61h3m55.900795631s) [noise=true , address=77.90.31.17:4770 , piece_id=JBSOJLCE23ZM2OSGNLCJASWHOAXC3YREJ4ENCEASHP5I5UYWC5GQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: <nil> Jul 27 00:40:11.165533280 (61h4m20.88809632s) [noise=true , address=104.36.82.16:4295 , piece_id=YCYD4ZOMTZ7XTBQATOXOCP7BSTSXS4P6BJK5KK47NONTTSA57BWA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165602982 (61h4m20.888166022s) [noise=true , address=104.36.82.8:4972 , piece_id=XMBGQ5KIOEKJQI3DK2UUHUKC663J2IYEHBSA6W3IKPXQDBUWTF3A, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165659470 (61h4m20.88822251s) [noise=true , address=77.90.31.15:4699 , piece_id=XDTFENIAZRV5QPJOGGKTHLACX6GS4K5AVDP5CIVAB46OVOOZXRAA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165747348 (61h4m20.888310388s) [noise=true , address=104.36.82.10:4079 , piece_id=OQCJDBFC33NB7AQLCTWOF5T3WSBCYXTAXW7YNJY74CE347YN4VCA, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165843924 (61h4m20.888406964s) [noise=true , address=104.36.82.2:4731 , piece_id=UFQNMT5PCF6IY5OD6MIKNF7K4CZBTSB2IQDETJ4QJ3SPTSR4N6TQ, seg_pos={0 0}, upload=58553]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:33372->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:11.165864824 (61h4m20.888427864s) [seg_pos={0 0}, upload=58553]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:40:11.394295096 (61h4m21.116858186s) [upload=58553]: Done uploading segments... Jul 27 00:40:11.394328439 (61h4m21.116891519s) [upload=58553]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.231908392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.240808854s FAIL (2025-07-27 00:40:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 96 (61h3m13.006453136s) [seg_pos={0 0}, upload=58546]: Begin upload segment... Jul 27 00:39:03.283954238 (61h3m13.006517288s) [seg_pos={0 0}, upload=58546]: Done begin upload segment. Jul 27 00:39:03.283964557 (61h3m13.006527678s) [noise=true , address=79.127.205.234:4612 , piece_id=7EJPSP6UGSBXD2L7IO5NUDSLZ3Q4L6VWCL3HH2UX2JZ3MWHGPMXA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.283976491 (61h3m13.006539531s) [noise=true , address=64.49.13.140:4508 , piece_id=5NJSO6573FLHESKZ7FBEOJWISL4VCIKQEPTVCLV2CFW355JUJBWA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.284391785 (61h3m13.006954825s) [noise=true , address=104.36.82.47:4548 , piece_id=TTHWZQZJNYW37MNGI2TLXI3HHIJPLECEV72CETXWLU3XJAZSANHA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.284689966 (61h3m13.007253006s) [noise=true , address=104.36.82.56:4247 , piece_id=JSZJHOX2FNB6MINTYPEEWGB2SISWZZHBU3FMRUY4HXJDD7L5GN6Q, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.284897492 (61h3m13.007460542s) [noise=true , address=64.49.13.138:4482 , piece_id=7O5XVSB5TRM4KRLYDP3P5E52VWRERWKMAYPK4DQBLTZUVXSX466Q, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285010729 (61h3m13.007573769s) [noise=true , address=77.90.31.12:4597 , piece_id=LAAI66KTPINHH4QQVVWTH56KM4UHG7WZCY5SEWZLDSLPQGYIPPXA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285174092 (61h3m13.007737122s) [noise=true , address=104.36.82.46:4489 , piece_id=LVIKRAOP2PE3FZPMQPUKBGK5OS4AQSDEHKN3LDMT7GOPVNBIGACA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285316995 (61h3m13.007880035s) [noise=true , address=104.36.82.37:4685 , piece_id=QXVTM2B6RG6HBSDBBEGBRQXKNKPTII37RJM2OXLQDOFEJ3H7WUFA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285555291 (61h3m13.008118331s) [noise=true , address=77.90.31.18:4833 , piece_id=FNSNVHPXYAKYY7VMER4V55Q7W3NT4IDCRR3537DLJEBVRIHJII6Q, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.285921642 (61h3m13.008484682s) [noise=true , address=79.127.205.235:4669 , piece_id=HR37H5QGSOIM3DGGOBTNUG3VVBR3JQLMV46VPPUGWK7FCYYOKKGQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.286310306 (61h3m13.008873346s) [noise=true , address=79.127.205.233:4624 , piece_id=6T4S7Q2FGJNULWMYK3BD244FW24EKICU2CKSE3KRR6AJNGVTRSTQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.286476755 (61h3m13.009039805s) [noise=true , address=77.90.31.11:4565 , piece_id=HF4PPC2URIELIUOFOPLVJHYDNQ5ZMSV3OD56XKL62GAAK2N7T4RQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.286813248 (61h3m13.009376298s) [noise=true , address=104.36.82.54:4179 , piece_id=GSSXZ5XTWYDYM43WNGC3OBF7HIWTB7AANQBTF3SAL44VCP67HIMA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.287136868 (61h3m13.009699908s) [noise=true , address=104.36.82.13:4217 , piece_id=MPLNAZQJ2FSLK65JCTZZ5BXWBLUTJIJR7LAJSMBGVAWLCCWKZS4A, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.287234844 (61h3m13.009797885s) [noise=true , address=104.36.82.52:4093 , piece_id=WU2NHJSSH5HMKGKR3ZPS2O7L7BHGOLRXVO4NY5NAMDEY2KHX3KOA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.287586958 (61h3m13.010150008s) [noise=true , address=79.127.205.246:4277 , piece_id=DIM3L5H663A6N6D6QMQQGHDGYGWVDP4VB5C43JDUJRFVJ2IH2ELA, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.287922600 (61h3m13.01048564s) [noise=true , address=79.127.220.101:4041 , piece_id=KDCVMSDLMI7MLBE2O5IZTDKYLC6QWJMPCOC32U4X2QA2XXKWOJTQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:03.288278451 (61h3m13.010841491s) [noise=true , address=45.91.116.14:4206 , piece_id=W43PZ4MVKZCEIBE55Z6I2FGDAG7G5T2DQO3UYO7WSOIRAPGRTLVQ, seg_pos={0 0}, upload=58546]: Uploading piece... Jul 27 00:39:04.388954830 (61h3m14.11151788s) [upload=58546]: Next returned nil segment Jul 27 00:39:04.388981491 (61h3m14.111544531s) [upload=58546]: Waiting for error group managing segments... Jul 27 00:39:04.685103236 (61h3m14.407666277s) [noise=true , address=79.127.205.234:4612 , piece_id=7EJPSP6UGSBXD2L7IO5NUDSLZ3Q4L6VWCL3HH2UX2JZ3MWHGPMXA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:04.689990508 (61h3m14.412553548s) [noise=true , address=79.127.205.235:4669 , piece_id=HR37H5QGSOIM3DGGOBTNUG3VVBR3JQLMV46VPPUGWK7FCYYOKKGQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:04.692098300 (61h3m14.41466134s) [noise=true , address=79.127.205.246:4277 , piece_id=DIM3L5H663A6N6D6QMQQGHDGYGWVDP4VB5C43JDUJRFVJ2IH2ELA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:04.706594225 (61h3m14.429157275s) [noise=true , address=79.127.205.233:4624 , piece_id=6T4S7Q2FGJNULWMYK3BD244FW24EKICU2CKSE3KRR6AJNGVTRSTQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:05.101755451 (61h3m14.824318491s) [noise=true , address=64.49.13.140:4508 , piece_id=5NJSO6573FLHESKZ7FBEOJWISL4VCIKQEPTVCLV2CFW355JUJBWA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:05.304648493 (61h3m15.027211523s) [noise=true , address=77.90.31.18:4833 , piece_id=FNSNVHPXYAKYY7VMER4V55Q7W3NT4IDCRR3537DLJEBVRIHJII6Q, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:05.347548484 (61h3m15.070111524s) [noise=true , address=77.90.31.11:4565 , piece_id=HF4PPC2URIELIUOFOPLVJHYDNQ5ZMSV3OD56XKL62GAAK2N7T4RQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:05.408810325 (61h3m15.131373365s) [noise=true , address=45.91.116.14:4206 , piece_id=W43PZ4MVKZCEIBE55Z6I2FGDAG7G5T2DQO3UYO7WSOIRAPGRTLVQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:07.062675451 (61h3m16.785238491s) [noise=true , address=77.90.31.12:4597 , piece_id=LAAI66KTPINHH4QQVVWTH56KM4UHG7WZCY5SEWZLDSLPQGYIPPXA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:07.433784748 (61h3m17.156347788s) [noise=true , address=79.127.220.101:4041 , piece_id=KDCVMSDLMI7MLBE2O5IZTDKYLC6QWJMPCOC32U4X2QA2XXKWOJTQ, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:39:12.527684124 (61h3m22.250247154s) [noise=true , address=64.49.13.138:4482 , piece_id=7O5XVSB5TRM4KRLYDP3P5E52VWRERWKMAYPK4DQBLTZUVXSX466Q, seg_pos={0 0}, upload=58546]: Done uploading piece. err: <nil> Jul 27 00:40:02.190434180 (61h4m11.91299723s) [noise=true , address=104.36.82.47:4548 , piece_id=TTHWZQZJNYW37MNGI2TLXI3HHIJPLECEV72CETXWLU3XJAZSANHA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190541675 (61h4m11.913104726s) [noise=true , address=104.36.82.56:4247 , piece_id=JSZJHOX2FNB6MINTYPEEWGB2SISWZZHBU3FMRUY4HXJDD7L5GN6Q, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: rpc: dial tcp 104.36.82.56:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190605237 (61h4m11.913168287s) [noise=true , address=104.36.82.54:4179 , piece_id=GSSXZ5XTWYDYM43WNGC3OBF7HIWTB7AANQBTF3SAL44VCP67HIMA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: write tcp 188.166.225.170:53772->104.36.82.54:4179: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190653319 (61h4m11.913216359s) [noise=true , address=104.36.82.46:4489 , piece_id=LVIKRAOP2PE3FZPMQPUKBGK5OS4AQSDEHKN3LDMT7GOPVNBIGACA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: rpc: dial tcp 104.36.82.46:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190669720 (61h4m11.91323277s) [noise=true , address=104.36.82.37:4685 , piece_id=QXVTM2B6RG6HBSDBBEGBRQXKNKPTII37RJM2OXLQDOFEJ3H7WUFA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12MhzeNvbKYhZVnhQW8oY5BeGQDuyNtmi25Xhgzrej4bCyoD9VM, address:104.36.82.37:4685): protocol: rpc: dial tcp 104.36.82.37:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190718413 (61h4m11.913281453s) [noise=true , address=104.36.82.13:4217 , piece_id=MPLNAZQJ2FSLK65JCTZZ5BXWBLUTJIJR7LAJSMBGVAWLCCWKZS4A, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:104.36.82.13:4217): protocol: rpc: dial tcp 104.36.82.13:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190768619 (61h4m11.913331669s) [noise=true , address=104.36.82.52:4093 , piece_id=WU2NHJSSH5HMKGKR3ZPS2O7L7BHGOLRXVO4NY5NAMDEY2KHX3KOA, seg_pos={0 0}, upload=58546]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: write tcp 188.166.225.170:32968->104.36.82.52:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:02.190797224 (61h4m11.913360535s) [seg_pos={0 0}, upload=58546]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:40:02.429216719 (61h4m12.151779769s) [upload=58546]: Done uploading segments... Jul 27 00:40:02.429248249 (61h4m12.151811289s) [upload=58546]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.240825697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X...] duration: 34.184643634s FAIL (2025-07-27 00:39:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.184654905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..] duration: 7.911199827s FAIL (2025-07-27 00:39:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.911219945s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX.....] duration: 6.504881858s FAIL (2025-07-27 00:39:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.504907338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXX] duration: 30.235322881s FAIL (2025-07-27 00:39:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tore close: conn closed: op Jul 27 00:38:26.756503444 (61h2m36.479066494s) [noise=true , address=104.36.82.6:4874 , piece_id=TX75MC4GTIOC3L6DLUTHORT4DWORHYTSKQQCNVZPUSVRWONZM5KA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:35416->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756520356 (61h2m36.479083386s) [seg_pos={0 0}, upload=58529]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:38:26.982864478 (61h2m36.705427518s) [upload=58529]: Done uploading segments... Jul 27 00:38:26.982888764 (61h2m36.705451804s) [upload=58529]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:39:00.518757775 (61h3m10.241320815s) [upload=58544]: Starting upload Jul 27 00:39:00.518861763 (61h3m10.241424813s) [upload=58544]: Done starting upload Jul 27 00:39:00.518978487 (61h3m10.241541527s) [upload=58544]: Uploading segments... Jul 27 00:39:00.519190152 (61h3m10.241753192s) [upload=58544]: Got next segment. Inline: false Jul 27 00:39:00.519268151 (61h3m10.241831201s) [upload=58544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:39:00.519274303 (61h3m10.241837363s) [upload=58544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:39:00.772196596 (61h3m10.494759646s) [seg_pos={0 0}, upload=58544]: Begin upload segment... Jul 27 00:39:00.772257002 (61h3m10.494820042s) [seg_pos={0 0}, upload=58544]: Done begin upload segment. Jul 27 00:39:00.772278513 (61h3m10.494841553s) [noise=true , address=77.90.31.12:4590 , piece_id=TDK7WYMWOSNAEQW55ENFSZY5P6UCAUCF7BW6SZ26UCFDI3KM2H7Q, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.772292560 (61h3m10.4948556s) [noise=true , address=45.91.116.13:4154 , piece_id=ACG7XISQPIIM7BZWJ2FXZAM577TXZNIRRIAKKAEWVFYHIWTSLQ7Q, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.772350882 (61h3m10.494913922s) [noise=true , address=45.91.116.6:4973 , piece_id=LR4FNOBTP2GN2FAFPJIKONKH32CHK2UWZ4XQFJ5NAHNCAZLZBSJA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.772823256 (61h3m10.495386296s) [noise=true , address=45.91.116.9:4263 , piece_id=BCHXOV7YM7GNNWTJXGQNQ56GHTNAJQ6XL3APJFYNAVND4OZTX3HQ, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.772871358 (61h3m10.495434388s) [noise=true , address=104.36.82.4:4804 , piece_id=LSZ2OOIYJQPQSXE34PHR3FA5EZW7CZ6HQJYPKYBN3CQLKJEU4DLQ, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.773245774 (61h3m10.495808824s) [noise=true , address=45.91.116.9:4262 , piece_id=5LBV4SRCIDQYFC6LPF2ZEBK2GJODZ57YCYHIFZSFK3EDTSE6LHBA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.773337259 (61h3m10.495900299s) [noise=true , address=77.90.31.14:4663 , piece_id=RW6KFZSSRJDGEDYWEVXDQUBHVRQTVUWBNNSHJGKELNNZ2H4XOZQA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.773618197 (61h3m10.496181237s) [noise=true , address=77.90.31.19:4843 , piece_id=FRDM2XKZ3RWV224KIACONFUZ2TLULLWUM25Q5EI4NMCLO7R62LOA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.773695344 (61h3m10.496258384s) [noise=true , address=45.91.116.4:4938 , piece_id=KPM5PXZK4TKFYZ24ATHXIJQWTHJYHIQMNSHZGJ2SGXW7A657PYKA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.774118363 (61h3m10.496681413s) [noise=true , address=45.91.116.14:4190 , piece_id=SL5CNE3TFOIX5TMHYB6ZF7LT3PBKZWCUQ7NOKRHO6FX4EATXIRZA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.774497489 (61h3m10.497060529s) [noise=true , address=104.36.82.16:4293 , piece_id=5L3LGH75ABJLM6LG2NZYDIJQGRYNJ5AN2I3FDXO64FRYIMLVFBEA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.774897784 (61h3m10.497460824s) [noise=true , address=77.90.31.13:4626 , piece_id=SF5O3K3ZNRCBOIW5A3K6GCBPDXVQJ6NBY57GXFJSY6BNHDVZ7ACA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.775283312 (61h3m10.497846352s) [noise=true , address=45.91.116.10:4082 , piece_id=RUUBCAPVSVBUZLL3GIFJLV4I52SLZJNAZNUXQU3V3KMKX5YKILUA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.775643621 (61h3m10.498206661s) [noise=true , address=77.90.31.21:4879 , piece_id=N3HYMQAOXGHJRLR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.776021434 (61h3m10.498584484s) [noise=true , address=77.90.31.15:4699 , piece_id=PBREYVQWOB54Q7VQYLEF3FPDN6W4GX4L2BT6F4UOLVPXVD4KEIYQ, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:00.776339173 (61h3m10.498902213s) [noise=true , address=45.91.116.11:4118 , piece_id=WNW4NVJMKKGCOPD5EHV34LJ5MACPSTVS42C4KEA6VEXG6IAINZLA, seg_pos={0 0}, upload=58544]: Uploading piece... Jul 27 00:39:02.523350266 (61h3m12.245913306s) [upload=58544]: Next returned nil segment Jul 27 00:39:02.523372387 (61h3m12.245935427s) [upload=58544]: Waiting for error group managing segments... Jul 27 00:39:02.803441189 (61h3m12.526004229s) [noise=true , address=77.90.31.14:4663 , piece_id=RW6KFZSSRJDGEDYWEVXDQUBHVRQTVUWBNNSHJGKELNNZ2H4XOZQA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.840550059 (61h3m12.563113099s) [noise=true , address=45.91.116.14:4190 , piece_id=SL5CNE3TFOIX5TMHYB6ZF7LT3PBKZWCUQ7NOKRHO6FX4EATXIRZA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.869696104 (61h3m12.592259144s) [noise=true , address=45.91.116.11:4118 , piece_id=WNW4NVJMKKGCOPD5EHV34LJ5MACPSTVS42C4KEA6VEXG6IAINZLA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.879743457 (61h3m12.602306497s) [noise=true , address=45.91.116.13:4154 , piece_id=ACG7XISQPIIM7BZWJ2FXZAM577TXZNIRRIAKKAEWVFYHIWTSLQ7Q, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.885416381 (61h3m12.607979421s) [noise=true , address=45.91.116.10:4082 , piece_id=RUUBCAPVSVBUZLL3GIFJLV4I52SLZJNAZNUXQU3V3KMKX5YKILUA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.888715092 (61h3m12.611278132s) [noise=true , address=45.91.116.6:4973 , piece_id=LR4FNOBTP2GN2FAFPJIKONKH32CHK2UWZ4XQFJ5NAHNCAZLZBSJA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.898264905 (61h3m12.620827945s) [noise=true , address=45.91.116.4:4938 , piece_id=KPM5PXZK4TKFYZ24ATHXIJQWTHJYHIQMNSHZGJ2SGXW7A657PYKA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.956219137 (61h3m12.678782177s) [noise=true , address=45.91.116.9:4262 , piece_id=5LBV4SRCIDQYFC6LPF2ZEBK2GJODZ57YCYHIFZSFK3EDTSE6LHBA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:02.956593604 (61h3m12.679156644s) [noise=true , address=45.91.116.9:4263 , piece_id=BCHXOV7YM7GNNWTJXGQNQ56GHTNAJQ6XL3APJFYNAVND4OZTX3HQ, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:04.245404156 (61h3m13.967967197s) [noise=true , address=77.90.31.19:4843 , piece_id=FRDM2XKZ3RWV224KIACONFUZ2TLULLWUM25Q5EI4NMCLO7R62LOA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:04.249864841 (61h3m13.972427881s) [noise=true , address=77.90.31.13:4626 , piece_id=SF5O3K3ZNRCBOIW5A3K6GCBPDXVQJ6NBY57GXFJSY6BNHDVZ7ACA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:05.008337626 (61h3m14.730900666s) [noise=true , address=77.90.31.12:4590 , piece_id=TDK7WYMWOSNAEQW55ENFSZY5P6UCAUCF7BW6SZ26UCFDI3KM2H7Q, seg_pos={0 0}, upload=58544]: Done uploading piece. err: <nil> Jul 27 00:39:29.867376960 (61h3m39.58994s) [noise=true , address=104.36.82.4:4804 , piece_id=LSZ2OOIYJQPQSXE34PHR3FA5EZW7CZ6HQJYPKYBN3CQLKJEU4DLQ, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:47338->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867392059 (61h3m39.589955099s) [noise=true , address=77.90.31.15:4699 , piece_id=PBREYVQWOB54Q7VQYLEF3FPDN6W4GX4L2BT6F4UOLVPXVD4KEIYQ, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867398612 (61h3m39.589961652s) [noise=true , address=104.36.82.16:4293 , piece_id=5L3LGH75ABJLM6LG2NZYDIJQGRYNJ5AN2I3FDXO64FRYIMLVFBEA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:49496->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867422488 (61h3m39.589985528s) [noise=true , address=77.90.31.21:4879 , piece_id=N3HYMQAOXGHJRLR7Y7CINYPS64DWP4KPVIOILA4KHMLRW2JHM6WA, seg_pos={0 0}, upload=58544]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:56638->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:29.867442766 (61h3m39.590005806s) [seg_pos={0 0}, upload=58544]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:39:30.101162500 (61h3m39.82372554s) [upload=58544]: Done uploading segments... Jul 27 00:39:30.101188930 (61h3m39.82375197s) [upload=58544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235345144s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX....] duration: 6.349223489s FAIL (2025-07-27 00:39:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.349242905s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XX..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X..] duration: 49.999475584s FAIL (2025-07-27 00:39:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.999486144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.842763924s FAIL (2025-07-27 00:39:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.842795825s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX.] duration: 7.60010323s FAIL (2025-07-27 00:39:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.600132556s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238455521s FAIL (2025-07-27 00:39:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== ess=104.36.82.56:4232 , piece_id=BBGYLAUDFKIMNRMUFAP53WKAR2KKTNFEM33NMV6C34Q6WV7RKKTQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.813599578 (61h2m12.536162618s) [noise=true , address=64.49.13.139:4478 , piece_id=OA2RYNGH4GL43RTSIPYY5NVSGIWV7JLRTENC4BQMBYY2MOI5XYPA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.814356225 (61h2m12.536919255s) [noise=true , address=104.36.82.54:4180 , piece_id=X43ASH4NVEFA5E7H7WIHIVOCJRNYLEOUZ6SGXA3OGZ2HKMUX722Q, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.814637593 (61h2m12.537200633s) [noise=true , address=77.90.31.15:4727 , piece_id=JGQA7LS5QWQZFKAU7VEICWHX643IALQFAKXH3D3NHYG3WSTG2ZHA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.814940092 (61h2m12.537503132s) [noise=true , address=104.36.82.53:4121 , piece_id=Z3GZITDFNCZXGLYPV2ASPVNRSSWHP5PMKR55R7W4DLYMKB6P5FBQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.815231099 (61h2m12.537794139s) [noise=true , address=64.49.13.138:4482 , piece_id=AMQTARPP6LQ3RGFJWA2NZD25KNRUL2UVZ35SBJB3GSAVOF5D3LSQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.815514090 (61h2m12.53807712s) [noise=true , address=77.90.31.11:4574 , piece_id=MMNHJYAYE57UQRMOKKIL5JJOJHJMHSGZZO5XMXCSQCRTANFJX2YQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.816467815 (61h2m12.539030865s) [noise=true , address=104.36.82.13:4209 , piece_id=POVGOCUKLKHVDYTDICHN34JK7UUVXBBGXI6YKPALAB6WWAE4YZDA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.816825570 (61h2m12.53938862s) [noise=true , address=79.127.205.234:4610 , piece_id=H7BKX2PCNY4LWW4GIXP3N2VN4FT3IBCMANFWIQBQJQKVZLXXYPRA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.817121366 (61h2m12.539684396s) [noise=true , address=77.90.31.10:4534 , piece_id=7T4PYLVXFNYMKD6WUU2FBOE2WOLYJVBMHXA7J2MAC6SFFQGYXUSA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.817560326 (61h2m12.540123366s) [noise=true , address=79.127.205.246:4262 , piece_id=JVPKREGGUI7NAORUZERP6ZCSZ5YO5K5KR54X52WA4TLATNDXK6YA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.817926045 (61h2m12.540489085s) [noise=true , address=79.127.205.233:4655 , piece_id=AUDCFQYWG6V77VLICZ4CWCU3G3PX6FSDPR3OS4U5E4EXAI5DJ4MA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.818258391 (61h2m12.540821441s) [noise=true , address=104.36.82.46:4516 , piece_id=VL3UZT2SBIDBF74L6F2BA3LXQQNSPSQN7XXUCB5FSDGNJVVURCSQ, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.818839833 (61h2m12.541402873s) [noise=true , address=79.127.205.245:4242 , piece_id=Z2V4VN3EMYWYZPGKUMHJ4YNXM3DAGDUPSMEYQD74ZIUARIUZHQ5A, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.819111042 (61h2m12.541674082s) [noise=true , address=77.90.31.13:4634 , piece_id=KF4OEJ42UQ56MIYSJVE47DBFQDY2ZED3D5D22CGCZP4O67W7SGZA, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:02.819444530 (61h2m12.54200757s) [noise=true , address=79.127.219.38:4001 , piece_id=LF2H5R52QMJGXXA6ZBGPQS2RRKPIDEUDRPSDLMXYSGMBD32UMD4A, seg_pos={0 0}, upload=58531]: Uploading piece... Jul 27 00:38:03.892037167 (61h2m13.614600207s) [noise=true , address=77.90.31.13:4634 , piece_id=KF4OEJ42UQ56MIYSJVE47DBFQDY2ZED3D5D22CGCZP4O67W7SGZA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, address:77.90.31.13:4634): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:03.912708226 (61h2m13.635271266s) [upload=58531]: Next returned nil segment Jul 27 00:38:03.912726081 (61h2m13.635289111s) [upload=58531]: Waiting for error group managing segments... Jul 27 00:38:04.203079740 (61h2m13.92564278s) [noise=true , address=79.127.205.234:4610 , piece_id=H7BKX2PCNY4LWW4GIXP3N2VN4FT3IBCMANFWIQBQJQKVZLXXYPRA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.263993836 (61h2m13.986556876s) [noise=true , address=79.127.205.245:4242 , piece_id=Z2V4VN3EMYWYZPGKUMHJ4YNXM3DAGDUPSMEYQD74ZIUARIUZHQ5A, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.277880706 (61h2m14.000443746s) [noise=true , address=79.127.205.246:4262 , piece_id=JVPKREGGUI7NAORUZERP6ZCSZ5YO5K5KR54X52WA4TLATNDXK6YA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.593509724 (61h2m14.316072754s) [noise=true , address=79.127.205.233:4655 , piece_id=AUDCFQYWG6V77VLICZ4CWCU3G3PX6FSDPR3OS4U5E4EXAI5DJ4MA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.708002823 (61h2m14.430565853s) [noise=true , address=79.127.219.38:4001 , piece_id=LF2H5R52QMJGXXA6ZBGPQS2RRKPIDEUDRPSDLMXYSGMBD32UMD4A, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:04.801373607 (61h2m14.523936647s) [noise=true , address=77.90.31.11:4574 , piece_id=MMNHJYAYE57UQRMOKKIL5JJOJHJMHSGZZO5XMXCSQCRTANFJX2YQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:38:05.697093515 (61h2m15.419656555s) [noise=true , address=64.49.13.138:4482 , piece_id=AMQTARPP6LQ3RGFJWA2NZD25KNRUL2UVZ35SBJB3GSAVOF5D3LSQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:18.733661915 (61h2m28.456224995s) [noise=true , address=64.49.13.139:4478 , piece_id=OA2RYNGH4GL43RTSIPYY5NVSGIWV7JLRTENC4BQMBYY2MOI5XYPA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: <nil> Jul 27 00:39:01.728045200 (61h3m11.45060825s) [noise=true , address=77.90.31.10:4534 , piece_id=7T4PYLVXFNYMKD6WUU2FBOE2WOLYJVBMHXA7J2MAC6SFFQGYXUSA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:01.728566166 (61h3m11.451129206s) [noise=true , address=104.36.82.56:4232 , piece_id=BBGYLAUDFKIMNRMUFAP53WKAR2KKTNFEM33NMV6C34Q6WV7RKKTQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1knYU6KSZJCmdpXTRHXZmCsRJYfNGGQCLPEPkWkBgUdyvtSgJe, address:104.36.82.56:4232): protocol: rpc: dial tcp 104.36.82.56:4232: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728607806 (61h3m11.451170856s) [noise=true , address=77.90.31.15:4727 , piece_id=JGQA7LS5QWQZFKAU7VEICWHX643IALQFAKXH3D3NHYG3WSTG2ZHA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728643524 (61h3m11.451206564s) [noise=true , address=104.36.82.53:4121 , piece_id=Z3GZITDFNCZXGLYPV2ASPVNRSSWHP5PMKR55R7W4DLYMKB6P5FBQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: rpc: dial tcp 104.36.82.53:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728688470 (61h3m11.45125152s) [noise=true , address=104.36.82.46:4516 , piece_id=VL3UZT2SBIDBF74L6F2BA3LXQQNSPSQN7XXUCB5FSDGNJVVURCSQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:12FRYABJbf6iHTm3WiYcL192kFth6rHjrcMAS3ahF8ddm4ox6Uj, address:104.36.82.46:4516): protocol: rpc: dial tcp 104.36.82.46:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728723847 (61h3m11.451286887s) [noise=true , address=104.36.82.47:4520 , piece_id=DHTRSISHBBTWDK2DTURXLPFSXBZECIR24AVBW5X5YANNBJWJIZ7Q, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: rpc: dial tcp 104.36.82.47:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728760208 (61h3m11.451323248s) [noise=true , address=104.36.82.13:4209 , piece_id=POVGOCUKLKHVDYTDICHN34JK7UUVXBBGXI6YKPALAB6WWAE4YZDA, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:18W2j3VhAMCmWAuST1M1yLgCWBZP3hQwGkLUvFAMSzUmhLpVRc, address:104.36.82.13:4209): protocol: rpc: dial tcp 104.36.82.13:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728801165 (61h3m11.451364215s) [noise=true , address=104.36.82.54:4180 , piece_id=X43ASH4NVEFA5E7H7WIHIVOCJRNYLEOUZ6SGXA3OGZ2HKMUX722Q, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728841623 (61h3m11.451404663s) [noise=true , address=104.36.82.9:4039 , piece_id=6LQJYVPLRYHPY6ZCNNHYDMPBCGBSTGKK3QKS6M4IK7MFUWADCIFQ, seg_pos={0 0}, upload=58531]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:4039): protocol: write tcp 188.166.225.170:52344->104.36.82.9:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:01.728858355 (61h3m11.451421395s) [seg_pos={0 0}, upload=58531]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:39:01.963894737 (61h3m11.686457777s) [upload=58531]: Done uploading segments... Jul 27 00:39:01.963933591 (61h3m11.686496631s) [upload=58531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.238495137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX] duration: 30.231335524s FAIL (2025-07-27 00:38:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23135541s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.851054536s FAIL (2025-07-27 00:38:35 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.85108238s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XX..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X.] duration: 55.559956416s FAIL (2025-07-27 00:38:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 55.559975772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXX] duration: 30.227796764s FAIL (2025-07-27 00:38:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 27 00:37:25.753540791 (61h1m35.476103861s) [noise=true , address=104.36.82.9:4005 , piece_id=H4FF52C5IFCW23IZXVFLDUIPICDGITIOP3QOIRNUX4Q5HH4AY7EQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; context deadline exceeded Jul 27 00:37:25.753560599 (61h1m35.476123629s) [seg_pos={0 0}, upload=58512]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:37:25.982334529 (61h1m35.704897569s) [upload=58512]: Done uploading segments... Jul 27 00:37:25.982363554 (61h1m35.704926594s) [upload=58512]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:37:57.395119345 (61h2m7.117682375s) [upload=58529]: Starting upload Jul 27 00:37:57.395186974 (61h2m7.117750004s) [upload=58529]: Done starting upload Jul 27 00:37:57.395260806 (61h2m7.117823836s) [upload=58529]: Uploading segments... Jul 27 00:37:57.395377409 (61h2m7.117940449s) [upload=58529]: Got next segment. Inline: false Jul 27 00:37:57.395415763 (61h2m7.117978803s) [upload=58529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:37:57.395420442 (61h2m7.117983482s) [upload=58529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:37:57.838860108 (61h2m7.561423138s) [seg_pos={0 0}, upload=58529]: Begin upload segment... Jul 27 00:37:57.838903682 (61h2m7.561466712s) [seg_pos={0 0}, upload=58529]: Done begin upload segment. Jul 27 00:37:57.838926074 (61h2m7.561489114s) [noise=true , address=104.36.82.13:4186 , piece_id=FOWJ63ZEQLGVLC5UCVPHU2DNGY2POPR2XSH4VK6KORUWRFZNRHYA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839032508 (61h2m7.561595538s) [noise=true , address=45.91.116.5:4299 , piece_id=N3JVJ3VWTNEEML7PPGERQ632YVEY6R5DVCPJ7YOPCKJO5EEQLLGQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839290873 (61h2m7.561853913s) [noise=true , address=45.91.116.12:4225 , piece_id=7W5TMYBFLWMAHHCTZPZECF4KXLQ7M246ZVSTQ2FJNFZ4OWLF2HTQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839504170 (61h2m7.56206721s) [noise=true , address=45.91.116.9:4263 , piece_id=NRLM5DFHZL5RZM2DUIYXKDLOIMIBIZKPSXAUG3FCDYTCA4CF54WA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839588593 (61h2m7.562151633s) [noise=true , address=77.90.31.15:4698 , piece_id=Y4YNEMHV3VTRZUUZFSTZTHC2HIMFPYUTPYLRYZ6NCGOI34NQXPDQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839784357 (61h2m7.562347397s) [noise=true , address=104.36.82.6:4874 , piece_id=TX75MC4GTIOC3L6DLUTHORT4DWORHYTSKQQCNVZPUSVRWONZM5KA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.839928182 (61h2m7.562491212s) [noise=true , address=45.91.116.13:4153 , piece_id=3CJCYVMEM24WNJFMHEWPQB3HVPB7HWJQ22XTUFXJNCO7T7LBRPSQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840119398 (61h2m7.562682438s) [noise=true , address=45.91.116.10:4081 , piece_id=QXFA3TKUKIURB2POZVRMACPUFBT4GHJLLPLCCKEVGDZDZKOCFIBQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840189011 (61h2m7.562752051s) [noise=true , address=45.91.116.8:4045 , piece_id=TFKIFVA6FMBE63ILNB6QTJBHPLYL7PEKSTGZ3LWRFUST2RMNFPXA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840414092 (61h2m7.562977132s) [noise=true , address=77.90.31.11:4553 , piece_id=E4MGVVEF3CX6RCKIFMZ7GAMHHPJAPANEFSGQKEJ4SQW4IYKE7FOA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840429411 (61h2m7.562992451s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXDQGDHP4NI633USSPA2NW4SBKW4OGUG764B5EXA6AKUULYLEAQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840696472 (61h2m7.563259512s) [noise=true , address=77.90.31.12:4590 , piece_id=ZCZF62HABMW4EVT4OBMWCPFAFMHY6LPYM7QJVFOBAH2HHIZT2UMA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840701151 (61h2m7.563264191s) [noise=true , address=77.90.31.14:4663 , piece_id=KE3ZNOFT3YI7FOVFPNHNBAMSF7NMUFYR23QEU7TXZNURLBD3MEFQ, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840940930 (61h2m7.56350397s) [noise=true , address=45.91.116.4:4939 , piece_id=YTQBXZKLMAB7KGB7BMYM7TA6J4NVMU6N7GUTAIK77DLRGY2D2A6A, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.840974193 (61h2m7.563537223s) [noise=true , address=45.91.116.11:4117 , piece_id=ZPVFASEDZOJ7PH2NOB3S4NIRH5TDW7OGTJUOLCQCZAAPGKUB35LA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:57.841186480 (61h2m7.56374952s) [noise=true , address=104.36.82.2:4732 , piece_id=O5UZK5W5HRROTFV45TPI35OLW2XLSVO4PPVCHQZVCNJYTMZ4YFDA, seg_pos={0 0}, upload=58529]: Uploading piece... Jul 27 00:37:59.572735848 (61h2m9.295298888s) [upload=58529]: Next returned nil segment Jul 27 00:37:59.572757530 (61h2m9.29532056s) [upload=58529]: Waiting for error group managing segments... Jul 27 00:37:59.849447747 (61h2m9.572010787s) [noise=true , address=45.91.116.13:4153 , piece_id=3CJCYVMEM24WNJFMHEWPQB3HVPB7HWJQ22XTUFXJNCO7T7LBRPSQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.849732161 (61h2m9.572295191s) [noise=true , address=77.90.31.14:4663 , piece_id=KE3ZNOFT3YI7FOVFPNHNBAMSF7NMUFYR23QEU7TXZNURLBD3MEFQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.855079944 (61h2m9.577642984s) [noise=true , address=45.91.116.12:4225 , piece_id=7W5TMYBFLWMAHHCTZPZECF4KXLQ7M246ZVSTQ2FJNFZ4OWLF2HTQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.934575471 (61h2m9.657138511s) [noise=true , address=45.91.116.10:4081 , piece_id=QXFA3TKUKIURB2POZVRMACPUFBT4GHJLLPLCCKEVGDZDZKOCFIBQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.937599247 (61h2m9.660162277s) [noise=true , address=77.90.31.11:4553 , piece_id=E4MGVVEF3CX6RCKIFMZ7GAMHHPJAPANEFSGQKEJ4SQW4IYKE7FOA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.944012296 (61h2m9.666575337s) [noise=true , address=45.91.116.9:4263 , piece_id=NRLM5DFHZL5RZM2DUIYXKDLOIMIBIZKPSXAUG3FCDYTCA4CF54WA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:37:59.990566447 (61h2m9.713129477s) [noise=true , address=45.91.116.8:4045 , piece_id=TFKIFVA6FMBE63ILNB6QTJBHPLYL7PEKSTGZ3LWRFUST2RMNFPXA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:00.008214494 (61h2m9.730777544s) [noise=true , address=45.91.116.4:4939 , piece_id=YTQBXZKLMAB7KGB7BMYM7TA6J4NVMU6N7GUTAIK77DLRGY2D2A6A, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:00.053756088 (61h2m9.776319138s) [noise=true , address=45.91.116.11:4117 , piece_id=ZPVFASEDZOJ7PH2NOB3S4NIRH5TDW7OGTJUOLCQCZAAPGKUB35LA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:00.099507814 (61h2m9.822070884s) [noise=true , address=45.91.116.5:4299 , piece_id=N3JVJ3VWTNEEML7PPGERQ632YVEY6R5DVCPJ7YOPCKJO5EEQLLGQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:01.567939563 (61h2m11.290502604s) [noise=true , address=77.90.31.12:4590 , piece_id=ZCZF62HABMW4EVT4OBMWCPFAFMHY6LPYM7QJVFOBAH2HHIZT2UMA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: <nil> Jul 27 00:38:26.756358587 (61h2m36.478921627s) [noise=true , address=104.36.82.2:4732 , piece_id=O5UZK5W5HRROTFV45TPI35OLW2XLSVO4PPVCHQZVCNJYTMZ4YFDA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756404715 (61h2m36.478967755s) [noise=true , address=77.90.31.15:4698 , piece_id=Y4YNEMHV3VTRZUUZFSTZTHC2HIMFPYUTPYLRYZ6NCGOI34NQXPDQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756436546 (61h2m36.478999586s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXDQGDHP4NI633USSPA2NW4SBKW4OGUG764B5EXA6AKUULYLEAQ, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756470811 (61h2m36.479033851s) [noise=true , address=104.36.82.13:4186 , piece_id=FOWJ63ZEQLGVLC5UCVPHU2DNGY2POPR2XSH4VK6KORUWRFZNRHYA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756503444 (61h2m36.479066494s) [noise=true , address=104.36.82.6:4874 , piece_id=TX75MC4GTIOC3L6DLUTHORT4DWORHYTSKQQCNVZPUSVRWONZM5KA, seg_pos={0 0}, upload=58529]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:35416->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:26.756520356 (61h2m36.479083386s) [seg_pos={0 0}, upload=58529]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:38:26.982864478 (61h2m36.705427518s) [upload=58529]: Done uploading segments... Jul 27 00:38:26.982888764 (61h2m36.705451804s) [upload=58529]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.227817254s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX...] duration: 7.354361816s FAIL (2025-07-27 00:38:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.354385191s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXX] duration: 30.24140986s FAIL (2025-07-27 00:38:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.241437193s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236805445s FAIL (2025-07-27 00:38:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== CPCXNZT353J6BFDAZRY7E3FMGZZFJA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.371291336 (61h1m12.093854376s) [noise=true , address=79.127.205.235:4662 , piece_id=KVNXPF65ZPGYW4JGBJILVDIRBM3OKNBMWBWHKHZJO7N6AXA2CA2A, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.371592963 (61h1m12.094156003s) [noise=true , address=79.127.205.246:4258 , piece_id=LT7IFH6N2J4PY2OMU7PDXYOT25Q7P7X5QPJEURBPRPPBB7CXORZA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.371694417 (61h1m12.094257457s) [noise=true , address=77.90.31.16:4766 , piece_id=5MICPG3KLZAM6DI6W7H2YMRESTWBEQCVGMEGFDV4B6SPGEIOVTFA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.371834706 (61h1m12.094397746s) [noise=true , address=104.36.82.52:4102 , piece_id=47QXMWNPJLWVTXES2B4LEUNI3HTOJ4COJP32TOINJT5AU4DBKNRA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372017967 (61h1m12.094581007s) [noise=true , address=45.91.116.9:4277 , piece_id=ODGEJBFLQWQX6SDVJHZHEMZZD6C5BQHQGX7U3CUJV3P7XVFWDSEA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372110914 (61h1m12.094673964s) [noise=true , address=104.36.82.46:4502 , piece_id=LXCIOQEKNVPBYHR65OKHQMBRBFWFL7EPU4S3NOBS4AYWTLVLNOFA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372272053 (61h1m12.094835093s) [noise=true , address=77.90.31.13:4659 , piece_id=5UZGQZOUA3JNHDEKYEIVTDCFCZHL6ZXLEDQ42G72ZW37PFZVV6RQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372384097 (61h1m12.094947147s) [noise=true , address=79.127.219.38:4001 , piece_id=JRLPBPI3XKWDWGASGVC3TK7KZ5JK7VZNMKXBS7LHXXB4Z56GRUNQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372512643 (61h1m12.095075683s) [noise=true , address=45.91.116.11:4130 , piece_id=ND3PCYSBUZW7BRVBR3ENGIOAUKFUZ2G4U3PT35376UTYZLWBEVFQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.372731301 (61h1m12.095294342s) [noise=true , address=156.146.43.225:4461 , piece_id=NAMVLJXO22XN3ALJOBQ3ZIA4RPZRQV2TAI5BU2YS7R75765GIK7A, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.373009924 (61h1m12.095572975s) [noise=true , address=104.36.82.6:4897 , piece_id=634ESXLDRPWNR76NDDV7CQ6RWGIA4KEECQ57JCA5SRHSDZBN24MQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.373279280 (61h1m12.09584232s) [noise=true , address=104.36.82.55:4196 , piece_id=KANVAUP5O7ESWJSN7FY2RWFDZZV4JKFEXBUIGTZ766BWO6HNHFGA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.373519629 (61h1m12.096082669s) [noise=true , address=104.36.82.42:4824 , piece_id=4PSTOFHG3OYPJYJWSWWUVYNDMBMD6IBM7QVAM2YAB2HMYOACNU7A, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.373763376 (61h1m12.096326416s) [noise=true , address=77.90.31.10:4535 , piece_id=6BUGLJUZBLBMNS7TQPZVNWWZ7HNBPNT4BMTBHGXIZ34AFPYNQGJQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.374040696 (61h1m12.096603736s) [noise=true , address=104.36.82.47:4548 , piece_id=DZCJFKR5D4QJMHCQOPK4RPYFUIHB5Z2XPAW6ODZWANG5TCHLJ2OQ, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.374283070 (61h1m12.09684612s) [noise=true , address=77.90.31.11:4579 , piece_id=L537CARZN4TF4SP4A7FDCOKZPR2ZQF63L77BT464FK2T3FZQQPIA, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:02.374522789 (61h1m12.097085829s) [noise=true , address=104.36.82.53:4132 , piece_id=LPW3JBPUYJD3VBMYTFFKH6PPKJG2GKCKWCW4D4VIILR5SLDXA35A, seg_pos={0 0}, upload=58514]: Uploading piece... Jul 27 00:37:03.391908288 (61h1m13.114471328s) [noise=true , address=156.146.43.225:4461 , piece_id=NAMVLJXO22XN3ALJOBQ3ZIA4RPZRQV2TAI5BU2YS7R75765GIK7A, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:1D4k7wnNtivVkHko5CSBuZH6zqMuFyfFfshkZwEPqnmpLads49, address:156.146.43.225:4461): protocol: storage node overloaded, request limit: 2000 Jul 27 00:37:03.394174424 (61h1m13.116737464s) [noise=true , address=77.90.31.13:4659 , piece_id=5UZGQZOUA3JNHDEKYEIVTDCFCZHL6ZXLEDQ42G72ZW37PFZVV6RQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: storage node overloaded, request limit: 2000 Jul 27 00:37:03.445295333 (61h1m13.167858363s) [noise=true , address=77.90.31.10:4535 , piece_id=6BUGLJUZBLBMNS7TQPZVNWWZ7HNBPNT4BMTBHGXIZ34AFPYNQGJQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: storage node overloaded, request limit: 2000 Jul 27 00:37:03.487138281 (61h1m13.209701321s) [upload=58514]: Next returned nil segment Jul 27 00:37:03.487155714 (61h1m13.209718754s) [upload=58514]: Waiting for error group managing segments... Jul 27 00:37:03.803770128 (61h1m13.526333168s) [noise=true , address=79.127.205.235:4662 , piece_id=KVNXPF65ZPGYW4JGBJILVDIRBM3OKNBMWBWHKHZJO7N6AXA2CA2A, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:03.805587385 (61h1m13.528150425s) [noise=true , address=79.127.205.246:4258 , piece_id=LT7IFH6N2J4PY2OMU7PDXYOT25Q7P7X5QPJEURBPRPPBB7CXORZA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:04.359774673 (61h1m14.082337714s) [noise=true , address=77.90.31.16:4766 , piece_id=5MICPG3KLZAM6DI6W7H2YMRESTWBEQCVGMEGFDV4B6SPGEIOVTFA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:04.448912750 (61h1m14.17147579s) [noise=true , address=79.127.219.38:4001 , piece_id=JRLPBPI3XKWDWGASGVC3TK7KZ5JK7VZNMKXBS7LHXXB4Z56GRUNQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:04.514013877 (61h1m14.236576917s) [noise=true , address=77.90.31.11:4579 , piece_id=L537CARZN4TF4SP4A7FDCOKZPR2ZQF63L77BT464FK2T3FZQQPIA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:04.722635389 (61h1m14.445198469s) [noise=true , address=45.91.116.9:4277 , piece_id=ODGEJBFLQWQX6SDVJHZHEMZZD6C5BQHQGX7U3CUJV3P7XVFWDSEA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:14.313642296 (61h1m24.036205326s) [noise=true , address=45.91.116.11:4130 , piece_id=ND3PCYSBUZW7BRVBR3ENGIOAUKFUZ2G4U3PT35376UTYZLWBEVFQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:37:24.029047161 (61h1m33.751610201s) [noise=true , address=77.90.31.12:4616 , piece_id=Z262EKINXVBQ44VXHM4NFOCPCXNZT353J6BFDAZRY7E3FMGZZFJA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: <nil> Jul 27 00:38:01.262280310 (61h2m10.98484337s) [noise=true , address=104.36.82.53:4132 , piece_id=LPW3JBPUYJD3VBMYTFFKH6PPKJG2GKCKWCW4D4VIILR5SLDXA35A, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:01.262498608 (61h2m10.985061658s) [noise=true , address=104.36.82.42:4824 , piece_id=4PSTOFHG3OYPJYJWSWWUVYNDMBMD6IBM7QVAM2YAB2HMYOACNU7A, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12QTjnomQYFETHbaJBWxWEPo5yweArk68bKPeooZXpRZrpw1ACq, address:104.36.82.42:4824): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:38:01.262522544 (61h2m10.985085624s) [noise=true , address=104.36.82.47:4548 , piece_id=DZCJFKR5D4QJMHCQOPK4RPYFUIHB5Z2XPAW6ODZWANG5TCHLJ2OQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: write tcp 188.166.225.170:33332->104.36.82.47:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:01.262574223 (61h2m10.985137263s) [noise=true , address=104.36.82.55:4196 , piece_id=KANVAUP5O7ESWJSN7FY2RWFDZZV4JKFEXBUIGTZ766BWO6HNHFGA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: rpc: dial tcp 104.36.82.55:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:01.262620200 (61h2m10.98518324s) [noise=true , address=104.36.82.46:4502 , piece_id=LXCIOQEKNVPBYHR65OKHQMBRBFWFL7EPU4S3NOBS4AYWTLVLNOFA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: write tcp 188.166.225.170:50744->104.36.82.46:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:01.262635329 (61h2m10.985198379s) [noise=true , address=104.36.82.52:4102 , piece_id=47QXMWNPJLWVTXES2B4LEUNI3HTOJ4COJP32TOINJT5AU4DBKNRA, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:38:01.262712547 (61h2m10.985275607s) [noise=true , address=104.36.82.6:4897 , piece_id=634ESXLDRPWNR76NDDV7CQ6RWGIA4KEECQ57JCA5SRHSDZBN24MQ, seg_pos={0 0}, upload=58514]: Done uploading piece. err: ecclient: upload failed (node:12qtpRnAcGiHmdR6J6ysdwaLPvejnCvpKm6MggqsezhbZ14Z5Bt, address:104.36.82.6:4897): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:38:01.262732946 (61h2m10.985295986s) [seg_pos={0 0}, upload=58514]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:38:01.497391895 (61h2m11.219954935s) [upload=58514]: Done uploading segments... Jul 27 00:38:01.497424437 (61h2m11.219987487s) [upload=58514]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.236819251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX..] duration: 22.129969487s FAIL (2025-07-27 00:37:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.12999262s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..XX..XX.X.XX...X...............XX......X...XXX..XXX..X..X.X] duration: 1m0.00122038s FAIL (2025-07-27 00:37:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001245308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................X................] duration: 1.858463773s FAIL (2025-07-27 00:37:28 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.85850476s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXX] duration: 30.23069085s FAIL (2025-07-27 00:37:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=5L6NBZQC3UFLXPX4EHH23C3SK3HAW5AAW2EXCYOKL56RUNKTM35A, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:55.282591689 (61h1m5.005154719s) [noise=true , address=104.36.82.12:4150 , piece_id=AFI5POE2LKVDPVWDI7OJK3346AUQZCSEFTG46V4UJJWV6BQDYO5Q, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op Jul 27 00:36:55.282616637 (61h1m5.005179677s) [seg_pos={0 0}, upload=58501]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:36:55.525510482 (61h1m5.248073522s) [upload=58501]: Done uploading segments... Jul 27 00:36:55.525557102 (61h1m5.248120142s) [upload=58501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:36:56.392498939 (61h1m6.115061979s) [upload=58512]: Starting upload Jul 27 00:36:56.392550648 (61h1m6.115113688s) [upload=58512]: Done starting upload Jul 27 00:36:56.392602407 (61h1m6.115165447s) [upload=58512]: Uploading segments... Jul 27 00:36:56.392727386 (61h1m6.115290436s) [upload=58512]: Got next segment. Inline: false Jul 27 00:36:56.392777331 (61h1m6.115340371s) [upload=58512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:36:56.392781760 (61h1m6.1153448s) [upload=58512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:36:56.627999299 (61h1m6.350562339s) [seg_pos={0 0}, upload=58512]: Begin upload segment... Jul 27 00:36:56.628042671 (61h1m6.350605711s) [seg_pos={0 0}, upload=58512]: Done begin upload segment. Jul 27 00:36:56.628066487 (61h1m6.350629517s) [noise=true , address=45.91.116.11:4117 , piece_id=JCULBSIB6PTBVHZGO7FKUYSFMLEQKELOONMAQOKLRCDURJGPEDXA, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628071366 (61h1m6.350634406s) [noise=true , address=45.91.116.14:4191 , piece_id=EAM6OKUB2X4476FI34CFKKLPDUZUD2YOEFCW6NP5CECXHLTRE6XQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628402299 (61h1m6.350965339s) [noise=true , address=45.91.116.9:4262 , piece_id=L5FSTJS2CDWEZNKFY5A2GOLKVPE62EE5VFFYOBHGOW72VBMWAVXQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628425574 (61h1m6.350988614s) [noise=true , address=77.90.31.10:4519 , piece_id=P7N7UYQTO5RBPXHALMTNFYOR74QZ6Q44IDJRAHQ6NVTP7UZVYMKQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628672517 (61h1m6.351235547s) [noise=true , address=45.91.116.8:4047 , piece_id=QCGIAUYY37ONH5SRCACOILUGRVVGLCGNUVIVLFAKOES4GK3FH2QA, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628717453 (61h1m6.351280493s) [noise=true , address=45.91.116.6:4973 , piece_id=ALDGNEUOI2HIIRXR22R7OEH5IPZ2NIGKC3UFDBEJYFME6JPJQTIQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.628916373 (61h1m6.351479413s) [noise=true , address=45.91.116.12:4226 , piece_id=AVQHSRAPFYPCBEIUAYT4YDJG27HROZ4Y7SMDH44IYV3GULQCPBJQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629024931 (61h1m6.351587971s) [noise=true , address=45.91.116.9:4263 , piece_id=BPRWHLOQWUXBRQUQN74CH65LQYX3V2ZBRBDDX2J2ARVDJQQTZWQQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629166371 (61h1m6.351729401s) [noise=true , address=45.91.116.13:4154 , piece_id=6TKB3GMZ4GSYRWCPBZP4NAM5B5YKCLB3I4SYYT2T6W2MKGJRNCHQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629363038 (61h1m6.351926088s) [noise=true , address=104.36.82.6:4873 , piece_id=AT47NWH3OPO46WJBDPAX7SBVNQPEVQBDFESTVRYBDCBKS2RTXCUQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629446738 (61h1m6.352009778s) [noise=true , address=45.91.116.4:4937 , piece_id=CLDU4KN4RDF7MU4GFILV3K36OANM7I3MGS55JJVIIKFHH6H36DBQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629654796 (61h1m6.352217836s) [noise=true , address=104.36.82.9:4005 , piece_id=H4FF52C5IFCW23IZXVFLDUIPICDGITIOP3QOIRNUX4Q5HH4AY7EQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629696646 (61h1m6.352259686s) [noise=true , address=104.36.82.15:4258 , piece_id=UCUPLNJLKE5ST7DZ2EBPQDA2TH5E7J7GG7INICENTEIXX22F6GRQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629897721 (61h1m6.352460751s) [noise=true , address=45.91.116.10:4083 , piece_id=LRZFDCUGV2YX5Q7GNBXXF7NVWRZ6WMRDBTM7VENE3WBPW6WIGRHQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.629916967 (61h1m6.352480007s) [noise=true , address=77.90.31.21:4877 , piece_id=3DCP3VATQUXELXXCGQNHGC36Z45ZLFNWBFXBYAH4G6ZKNDKSVGGQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:56.630155995 (61h1m6.352719035s) [noise=true , address=45.91.116.4:4938 , piece_id=W775KX7CIUM4ALKYVFUWMTINTJ5TS76TECGTGPGMU2FKUJOMNGGQ, seg_pos={0 0}, upload=58512]: Uploading piece... Jul 27 00:36:58.359101342 (61h1m8.081664382s) [upload=58512]: Next returned nil segment Jul 27 00:36:58.359120808 (61h1m8.081683848s) [upload=58512]: Waiting for error group managing segments... Jul 27 00:36:58.636622779 (61h1m8.359185819s) [noise=true , address=45.91.116.8:4047 , piece_id=QCGIAUYY37ONH5SRCACOILUGRVVGLCGNUVIVLFAKOES4GK3FH2QA, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.718032168 (61h1m8.440595208s) [noise=true , address=45.91.116.4:4937 , piece_id=CLDU4KN4RDF7MU4GFILV3K36OANM7I3MGS55JJVIIKFHH6H36DBQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.718474775 (61h1m8.441037815s) [noise=true , address=45.91.116.12:4226 , piece_id=AVQHSRAPFYPCBEIUAYT4YDJG27HROZ4Y7SMDH44IYV3GULQCPBJQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.722680021 (61h1m8.445243061s) [noise=true , address=45.91.116.10:4083 , piece_id=LRZFDCUGV2YX5Q7GNBXXF7NVWRZ6WMRDBTM7VENE3WBPW6WIGRHQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.736511374 (61h1m8.459074414s) [noise=true , address=45.91.116.9:4262 , piece_id=L5FSTJS2CDWEZNKFY5A2GOLKVPE62EE5VFFYOBHGOW72VBMWAVXQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.736719141 (61h1m8.459282181s) [noise=true , address=45.91.116.6:4973 , piece_id=ALDGNEUOI2HIIRXR22R7OEH5IPZ2NIGKC3UFDBEJYFME6JPJQTIQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.782897574 (61h1m8.505460614s) [noise=true , address=45.91.116.14:4191 , piece_id=EAM6OKUB2X4476FI34CFKKLPDUZUD2YOEFCW6NP5CECXHLTRE6XQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.790103231 (61h1m8.512666271s) [noise=true , address=45.91.116.13:4154 , piece_id=6TKB3GMZ4GSYRWCPBZP4NAM5B5YKCLB3I4SYYT2T6W2MKGJRNCHQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.791579856 (61h1m8.514142886s) [noise=true , address=45.91.116.4:4938 , piece_id=W775KX7CIUM4ALKYVFUWMTINTJ5TS76TECGTGPGMU2FKUJOMNGGQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.804398562 (61h1m8.526961592s) [noise=true , address=45.91.116.9:4263 , piece_id=BPRWHLOQWUXBRQUQN74CH65LQYX3V2ZBRBDDX2J2ARVDJQQTZWQQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:36:58.809874008 (61h1m8.532437048s) [noise=true , address=45.91.116.11:4117 , piece_id=JCULBSIB6PTBVHZGO7FKUYSFMLEQKELOONMAQOKLRCDURJGPEDXA, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:37:00.167671442 (61h1m9.890234482s) [noise=true , address=77.90.31.10:4519 , piece_id=P7N7UYQTO5RBPXHALMTNFYOR74QZ6Q44IDJRAHQ6NVTP7UZVYMKQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: <nil> Jul 27 00:37:25.753434958 (61h1m35.475997998s) [noise=true , address=104.36.82.6:4873 , piece_id=AT47NWH3OPO46WJBDPAX7SBVNQPEVQBDFESTVRYBDCBKS2RTXCUQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:25.753452572 (61h1m35.476015612s) [noise=true , address=104.36.82.15:4258 , piece_id=UCUPLNJLKE5ST7DZ2EBPQDA2TH5E7J7GG7INICENTEIXX22F6GRQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op Jul 27 00:37:25.753478221 (61h1m35.476041261s) [noise=true , address=77.90.31.21:4877 , piece_id=3DCP3VATQUXELXXCGQNHGC36Z45ZLFNWBFXBYAH4G6ZKNDKSVGGQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:25.753540791 (61h1m35.476103861s) [noise=true , address=104.36.82.9:4005 , piece_id=H4FF52C5IFCW23IZXVFLDUIPICDGITIOP3QOIRNUX4Q5HH4AY7EQ, seg_pos={0 0}, upload=58512]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; context deadline exceeded Jul 27 00:37:25.753560599 (61h1m35.476123629s) [seg_pos={0 0}, upload=58512]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:37:25.982334529 (61h1m35.704897569s) [upload=58512]: Done uploading segments... Jul 27 00:37:25.982363554 (61h1m35.704926594s) [upload=58512]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230708824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XX] duration: 30.305544949s FAIL (2025-07-27 00:37:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.305568213s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX.] duration: 6.241579084s FAIL (2025-07-27 00:37:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.241594212s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230890006s FAIL (2025-07-27 00:37:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== TBZUKA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.889934156 (61h0m11.612497196s) [noise=true , address=104.36.82.46:4487 , piece_id=FT7FGBGJWRFT6VJL7WLYBWNDJPVLKWVWVIUBFQORPIDLYN2JTXTA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.890147744 (61h0m11.612710785s) [noise=true , address=104.36.82.47:4531 , piece_id=L5E2PLBEBCZKGZCFKHI2QUCWI6WE7R2WJWEOEK5VN2MNGJ472PYQ, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.890351183 (61h0m11.612914223s) [noise=true , address=104.36.82.52:4100 , piece_id=5YTBS5E6AB6GLY42Q4BXX5QV6NXDI5O7TCVXJXQ4OAJYLX4UJXIQ, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.890363306 (61h0m11.612926356s) [noise=true , address=104.36.82.34:4552 , piece_id=SKQGKND2BMTLSMUE3K4OT26CQNMDRHCX5UTKOV6ECNVIHR64PY7Q, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.890588047 (61h0m11.613151087s) [noise=true , address=104.36.82.55:4200 , piece_id=FBHEPWCNAEZKQSOJUAWDA77EEBTO6UATGJWQBXE2YJSJZ7FCWFTQ, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.890767359 (61h0m11.6133304s) [noise=true , address=104.36.82.41:4766 , piece_id=DOX4OUXRH54NLKPAVVA37NNZEZR6R67GKPSD3DAJTGULCSFSZVKA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.890807527 (61h0m11.613370567s) [noise=true , address=45.91.116.14:4206 , piece_id=VLMNNK66BPDEY2HHBUZ3AZ2XBJFZZW5YILVBD2CZLQQPLOQD2PWA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.891193175 (61h0m11.613756215s) [noise=true , address=104.36.82.53:4150 , piece_id=KKJ7ZE3ZIV3EZQTBNZMONGPBSLFQY4ADS6OOTYOQ6NPF7F5IY7CA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.891612417 (61h0m11.614175457s) [noise=true , address=79.127.220.97:4361 , piece_id=GU7WKHU5ITKROV44WL7BH2QPTRHAXY5G3IQMNXQFXIGQ3N2N3ZYA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.894005345 (61h0m11.616568395s) [noise=true , address=77.90.31.18:4832 , piece_id=JNX4HHWGEFFOEPSM5ZXT2YKNKEJZXW6QNG2YA2CHCE36X2MOJCCQ, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.895252601 (61h0m11.617815641s) [noise=true , address=79.127.219.38:4018 , piece_id=D3LKOQ6QSWQA6D6VTXYUJNKZYAFJ7GAUDV6DGVKDC7J3STWHJ3RA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.895482532 (61h0m11.618045572s) [noise=true , address=104.36.82.54:4174 , piece_id=JPOAQAFWYIDR35U3MGJTPH7ZWDCJJPIXXZMY3C5H45NVZKJQAEDA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.896196527 (61h0m11.618759567s) [noise=true , address=156.146.43.225:4449 , piece_id=7B6GQJCAGRJAJYL2YNWRGOBO2JEELHQ2WO36BPCPBYKYNSIJZKMA, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:01.897048557 (61h0m11.619611607s) [noise=true , address=79.127.219.47:4048 , piece_id=OY74J4SNFBQOF3FJY3TCZYCTZTGRIW6AS3AH5IXAFCM23UTI5Q4Q, seg_pos={0 0}, upload=58494]: Uploading piece... Jul 27 00:36:02.965254241 (61h0m12.687817281s) [noise=true , address=79.127.219.47:4048 , piece_id=OY74J4SNFBQOF3FJY3TCZYCTZTGRIW6AS3AH5IXAFCM23UTI5Q4Q, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:36:02.993859340 (61h0m12.71642238s) [upload=58494]: Next returned nil segment Jul 27 00:36:02.993869439 (61h0m12.716432469s) [upload=58494]: Waiting for error group managing segments... Jul 27 00:36:03.357753203 (61h0m13.080316243s) [noise=true , address=79.127.205.245:4218 , piece_id=2JAGB5YKR4Z7FJ23TCVKL4NFA5ILDXDJOOWYA2G6TPPJXRWMAQVA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: <nil> Jul 27 00:36:03.360747252 (61h0m13.083310352s) [noise=true , address=79.127.205.234:4612 , piece_id=XAJWHTDSGSDWMCZE4PVMH4WPCEQEXH6X4CE4NTJGXP4LITTBZUKA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: <nil> Jul 27 00:36:03.788614204 (61h0m13.511177234s) [noise=true , address=79.127.219.38:4018 , piece_id=D3LKOQ6QSWQA6D6VTXYUJNKZYAFJ7GAUDV6DGVKDC7J3STWHJ3RA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: <nil> Jul 27 00:36:04.324588055 (61h0m14.047151085s) [noise=true , address=77.90.31.10:4545 , piece_id=EAPZJESYDYUSYFUJ7TS6RTVKWXHFRYSZZTZCETUXTXF7G6STP53Q, seg_pos={0 0}, upload=58494]: Done uploading piece. err: <nil> Jul 27 00:36:04.616769111 (61h0m14.339332151s) [noise=true , address=79.127.220.97:4361 , piece_id=GU7WKHU5ITKROV44WL7BH2QPTRHAXY5G3IQMNXQFXIGQ3N2N3ZYA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: <nil> Jul 27 00:36:07.025191150 (61h0m16.74775418s) [noise=true , address=156.146.43.225:4449 , piece_id=7B6GQJCAGRJAJYL2YNWRGOBO2JEELHQ2WO36BPCPBYKYNSIJZKMA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: <nil> Jul 27 00:36:08.010042252 (61h0m17.732605292s) [noise=true , address=45.91.116.14:4206 , piece_id=VLMNNK66BPDEY2HHBUZ3AZ2XBJFZZW5YILVBD2CZLQQPLOQD2PWA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: <nil> Jul 27 00:37:00.795375339 (61h1m10.517938369s) [noise=true , address=104.36.82.52:4100 , piece_id=5YTBS5E6AB6GLY42Q4BXX5QV6NXDI5O7TCVXJXQ4OAJYLX4UJXIQ, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:104.36.82.52:4100): protocol: rpc: dial tcp 104.36.82.52:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795421087 (61h1m10.517984117s) [noise=true , address=104.36.82.34:4552 , piece_id=SKQGKND2BMTLSMUE3K4OT26CQNMDRHCX5UTKOV6ECNVIHR64PY7Q, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:12YmWSZZQgXm2gdZCahnyCdSzxYPAerzCfD38mWPmpTDyGYT823, address:104.36.82.34:4552): protocol: rpc: dial tcp 104.36.82.34:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795458509 (61h1m10.518021549s) [noise=true , address=77.90.31.18:4832 , piece_id=JNX4HHWGEFFOEPSM5ZXT2YKNKEJZXW6QNG2YA2CHCE36X2MOJCCQ, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: write tcp 188.166.225.170:60458->77.90.31.18:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795494277 (61h1m10.518057307s) [noise=true , address=104.36.82.54:4174 , piece_id=JPOAQAFWYIDR35U3MGJTPH7ZWDCJJPIXXZMY3C5H45NVZKJQAEDA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: write tcp 188.166.225.170:43316->104.36.82.54:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795507912 (61h1m10.518070953s) [noise=true , address=77.90.31.21:4907 , piece_id=YIIN7QLUW7CZ5W3IFDN3I6EAKBAVTO4KDH7P2LIMFNHACLWLT4XQ, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 188.166.225.170:38450->77.90.31.21:4907: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795545735 (61h1m10.518108776s) [noise=true , address=104.36.82.41:4766 , piece_id=DOX4OUXRH54NLKPAVVA37NNZEZR6R67GKPSD3DAJTGULCSFSZVKA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: write tcp 188.166.225.170:36118->104.36.82.41:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795585691 (61h1m10.518148732s) [noise=true , address=104.36.82.55:4200 , piece_id=FBHEPWCNAEZKQSOJUAWDA77EEBTO6UATGJWQBXE2YJSJZ7FCWFTQ, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795600130 (61h1m10.51816317s) [noise=true , address=104.36.82.53:4150 , piece_id=KKJ7ZE3ZIV3EZQTBNZMONGPBSLFQY4ADS6OOTYOQ6NPF7F5IY7CA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: write tcp 188.166.225.170:38472->104.36.82.53:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795609978 (61h1m10.518173018s) [noise=true , address=104.36.82.47:4531 , piece_id=L5E2PLBEBCZKGZCFKHI2QUCWI6WE7R2WJWEOEK5VN2MNGJ472PYQ, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795617162 (61h1m10.518180202s) [noise=true , address=104.36.82.46:4487 , piece_id=FT7FGBGJWRFT6VJL7WLYBWNDJPVLKWVWVIUBFQORPIDLYN2JTXTA, seg_pos={0 0}, upload=58494]: Done uploading piece. err: ecclient: upload failed (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX, address:104.36.82.46:4487): protocol: rpc: dial tcp 104.36.82.46:4487: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:00.795664242 (61h1m10.518227282s) [seg_pos={0 0}, upload=58494]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:37:01.024288486 (61h1m10.746851526s) [upload=58494]: Done uploading segments... Jul 27 00:37:01.024357167 (61h1m10.746920217s) [upload=58494]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.230903701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X..........................................X...............] duration: 1.925146635s FAIL (2025-07-27 00:36:59 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.925171464s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXX] duration: 30.244505232s FAIL (2025-07-27 00:36:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:35778->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767663897 (61h0m16.490226948s) [seg_pos={0 0}, upload=58486]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 27 00:36:06.997501503 (61h0m16.720064543s) [upload=58486]: Done uploading segments... Jul 27 00:36:06.997521811 (61h0m16.720084851s) [upload=58486]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Jul 27 00:36:25.930198760 (61h0m35.6527618s) [upload=58501]: Starting upload Jul 27 00:36:25.930249577 (61h0m35.652812617s) [upload=58501]: Done starting upload Jul 27 00:36:25.930418120 (61h0m35.65298116s) [upload=58501]: Uploading segments... Jul 27 00:36:25.930448077 (61h0m35.653011117s) [upload=58501]: Got next segment. Inline: false Jul 27 00:36:25.930469077 (61h0m35.653032117s) [upload=58501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:36:25.930473085 (61h0m35.653036125s) [upload=58501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:36:26.179199139 (61h0m35.901762169s) [seg_pos={0 0}, upload=58501]: Begin upload segment... Jul 27 00:36:26.179238756 (61h0m35.901801796s) [seg_pos={0 0}, upload=58501]: Done begin upload segment. Jul 27 00:36:26.179260517 (61h0m35.901823547s) [noise=true , address=104.36.82.12:4150 , piece_id=AFI5POE2LKVDPVWDI7OJK3346AUQZCSEFTG46V4UJJWV6BQDYO5Q, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.179269224 (61h0m35.901832264s) [noise=true , address=77.90.31.14:4663 , piece_id=3F5ZD6UEKBSEHBH6M6IS4KVPND3BHME2ZFBU7TAJQBWONRBJOHHQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.179603824 (61h0m35.902166864s) [noise=true , address=45.91.116.14:4191 , piece_id=GK37YTMNS5B5BBAQOLEOUPMQHVVC355IEN2476C3GZNTRJRBD6KQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.179724995 (61h0m35.902288035s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5ESTF5XP2QBNJL52XCKIUMFNGLHYQZQL7P6OAMV3RQ4JOI3I7TQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.179884711 (61h0m35.902447751s) [noise=true , address=104.36.82.8:4972 , piece_id=5L6NBZQC3UFLXPX4EHH23C3SK3HAW5AAW2EXCYOKL56RUNKTM35A, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.180061990 (61h0m35.90262503s) [noise=true , address=77.90.31.13:4627 , piece_id=RMD4EYTTKVV7WBCLDBIIV66IQJFZH7VM6ZMZRWXY46BPCIQLNM7A, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.180127857 (61h0m35.902690897s) [noise=true , address=77.90.31.16:4735 , piece_id=ORP4H66HDS36AFFPEJXRPGSN4QO3E4CXO2TXOF7C6N3I4D42AYCQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.180416007 (61h0m35.902979047s) [noise=true , address=104.36.82.13:4187 , piece_id=M344TTPQ6D77TJTJCUDT322MERBL3SPLTHJFIJRMP4ADSOX576DA, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.180486052 (61h0m35.903049092s) [noise=true , address=45.91.116.8:4048 , piece_id=APL7FDOWH47OJ2CMWXOEGR4OLCUR6TMYBVX5ILUOUDHSH3OZGCIQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.180660265 (61h0m35.903223305s) [noise=true , address=104.36.82.7:4911 , piece_id=VLUP3BSIHXUDUNJNPYHYHAFW5BZRDOSG2RYXP7CZBCHC2EY55DXQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.180818849 (61h0m35.903381889s) [noise=true , address=104.36.82.3:4765 , piece_id=47JZTRCA227Q3HO5HKUKSAYDSKPDIERJF4NRTUU3BBPXW4GMFB3A, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.180952995 (61h0m35.903516045s) [noise=true , address=45.91.116.9:4263 , piece_id=S6MGRTCCLNRLZYVWC75NTAQ57EOXCOFYZGUP6RZFURNMYYDFVGWA, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.181188776 (61h0m35.903751816s) [noise=true , address=45.91.116.13:4153 , piece_id=AC57PSKXANKELVOVMKBZPGJEI63NKEWT26SYTDIIP2536JXADCHA, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.181971514 (61h0m35.904534554s) [noise=true , address=77.90.31.12:4589 , piece_id=KZCRSM6K7ZGRWMPT7YY2B64ML2X6EKBJSHCSM2ZQW3OI3GKJ4SFQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.182205161 (61h0m35.904768191s) [noise=true , address=45.91.116.11:4119 , piece_id=RAKJECMTPFKLC64RVAGRF4WFGVKGWUECXGNYM4KWUFECLYMG52SQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:26.182413029 (61h0m35.904976069s) [noise=true , address=77.90.31.15:4697 , piece_id=H2RZQY2QDGH7EER3MJJJDS7GOCZGSO6J36BMBYEOVZHYEBDF4ZIQ, seg_pos={0 0}, upload=58501]: Uploading piece... Jul 27 00:36:27.905123989 (61h0m37.627687019s) [upload=58501]: Next returned nil segment Jul 27 00:36:27.905139539 (61h0m37.627702579s) [upload=58501]: Waiting for error group managing segments... Jul 27 00:36:28.191445333 (61h0m37.914008373s) [noise=true , address=45.91.116.9:4263 , piece_id=S6MGRTCCLNRLZYVWC75NTAQ57EOXCOFYZGUP6RZFURNMYYDFVGWA, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:28.202544560 (61h0m37.9251076s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5ESTF5XP2QBNJL52XCKIUMFNGLHYQZQL7P6OAMV3RQ4JOI3I7TQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:28.269993208 (61h0m37.992556238s) [noise=true , address=45.91.116.8:4048 , piece_id=APL7FDOWH47OJ2CMWXOEGR4OLCUR6TMYBVX5ILUOUDHSH3OZGCIQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:28.285032122 (61h0m38.007595162s) [noise=true , address=77.90.31.16:4735 , piece_id=ORP4H66HDS36AFFPEJXRPGSN4QO3E4CXO2TXOF7C6N3I4D42AYCQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:28.355520055 (61h0m38.078083095s) [noise=true , address=45.91.116.13:4153 , piece_id=AC57PSKXANKELVOVMKBZPGJEI63NKEWT26SYTDIIP2536JXADCHA, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:28.361589979 (61h0m38.084153019s) [noise=true , address=77.90.31.14:4663 , piece_id=3F5ZD6UEKBSEHBH6M6IS4KVPND3BHME2ZFBU7TAJQBWONRBJOHHQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:28.363166415 (61h0m38.085729455s) [noise=true , address=45.91.116.11:4119 , piece_id=RAKJECMTPFKLC64RVAGRF4WFGVKGWUECXGNYM4KWUFECLYMG52SQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:28.408191952 (61h0m38.130754992s) [noise=true , address=45.91.116.14:4191 , piece_id=GK37YTMNS5B5BBAQOLEOUPMQHVVC355IEN2476C3GZNTRJRBD6KQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:29.636326382 (61h0m39.358889422s) [noise=true , address=77.90.31.12:4589 , piece_id=KZCRSM6K7ZGRWMPT7YY2B64ML2X6EKBJSHCSM2ZQW3OI3GKJ4SFQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:29.760209185 (61h0m39.482772266s) [noise=true , address=77.90.31.13:4627 , piece_id=RMD4EYTTKVV7WBCLDBIIV66IQJFZH7VM6ZMZRWXY46BPCIQLNM7A, seg_pos={0 0}, upload=58501]: Done uploading piece. err: <nil> Jul 27 00:36:55.282347822 (61h1m5.004910873s) [noise=true , address=77.90.31.15:4697 , piece_id=H2RZQY2QDGH7EER3MJJJDS7GOCZGSO6J36BMBYEOVZHYEBDF4ZIQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:36:55.282366228 (61h1m5.004929268s) [noise=true , address=104.36.82.3:4765 , piece_id=47JZTRCA227Q3HO5HKUKSAYDSKPDIERJF4NRTUU3BBPXW4GMFB3A, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:41824->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:55.282409520 (61h1m5.00497256s) [noise=true , address=104.36.82.13:4187 , piece_id=M344TTPQ6D77TJTJCUDT322MERBL3SPLTHJFIJRMP4ADSOX576DA, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:55.282431202 (61h1m5.004994242s) [noise=true , address=104.36.82.7:4911 , piece_id=VLUP3BSIHXUDUNJNPYHYHAFW5BZRDOSG2RYXP7CZBCHC2EY55DXQ, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:36:55.282477952 (61h1m5.005040992s) [noise=true , address=104.36.82.8:4972 , piece_id=5L6NBZQC3UFLXPX4EHH23C3SK3HAW5AAW2EXCYOKL56RUNKTM35A, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:55.282591689 (61h1m5.005154719s) [noise=true , address=104.36.82.12:4150 , piece_id=AFI5POE2LKVDPVWDI7OJK3346AUQZCSEFTG46V4UJJWV6BQDYO5Q, seg_pos={0 0}, upload=58501]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op Jul 27 00:36:55.282616637 (61h1m5.005179677s) [seg_pos={0 0}, upload=58501]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:36:55.525510482 (61h1m5.248073522s) [upload=58501]: Done uploading segments... Jul 27 00:36:55.525557102 (61h1m5.248120142s) [upload=58501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.244528106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X..........................................X..............] duration: 1.846353773s FAIL (2025-07-27 00:36:42 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.846375515s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.X] duration: 30.23235479s FAIL (2025-07-27 00:36:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232379959s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X....XXX.X...X......X.XX...XXXX.....X..........X..X..X...XX] duration: 30.000499537s FAIL (2025-07-27 00:36:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000515707s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXX] duration: 30.232618311s FAIL (2025-07-27 00:36:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 362751258s) [noise=true , address=104.36.82.15:4258 , piece_id=ISCHVFRTDGDJMQV3PWQCXFTS6ESLL7SN472HWSS6LMHPV4CGRTAQ, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.640209369 (60h59m47.362772429s) [noise=true , address=104.36.82.14:4223 , piece_id=2DZEOIASHAVXTSJRCETE3R2MSYOCOZLWQJVXDBCDO2GFDPFALWVQ, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.640593714 (60h59m47.363156744s) [noise=true , address=104.36.82.2:4730 , piece_id=CYXYHS3NHHZ4LAXL4SWV4N4JOGPXGA3TSFAFNYWDNEBWO37SJAAQ, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.640760063 (60h59m47.363323113s) [noise=true , address=104.36.82.11:4114 , piece_id=T75LXCDLLJH5AAA3DZ5RMNTSPITHR357J2XQ4VAGWJF2UESBLURA, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.640952341 (60h59m47.363515391s) [noise=true , address=77.90.31.18:4805 , piece_id=M6BDVBEP3S5OL3UBJXZSV54T4UHSTEWAZCVXM2ZDA3VLDZVN3RVQ, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.641158205 (60h59m47.363721245s) [noise=true , address=104.36.82.5:4838 , piece_id=PM3E3G2AYXXBPGAJYUO3YBCAC27VQPN76JVPHTPM4ALYAEVHKBJQ, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.641617824 (60h59m47.364180884s) [noise=true , address=45.91.116.13:4154 , piece_id=GZ7YGJ2L4DLFQIK7OF6B6K5QJVEEPMKLLG7SEMNC2MUO72FAXU5A, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.642302033 (60h59m47.364865083s) [noise=true , address=77.90.31.17:4771 , piece_id=EE7CMF5HQUKVCWFHKATGYXTQCSL3EDP4YA4TZ7BL4I4YR75JVG2A, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.642728299 (60h59m47.365291359s) [noise=true , address=77.90.31.16:4733 , piece_id=NMMZOBY3S6E63IIWMB7QDBKNFP5PM2RODTZPJRJXPR5IKLQAX4CA, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.643069161 (60h59m47.365632191s) [noise=true , address=77.90.31.15:4697 , piece_id=WH23YMYXHCGEE6OAU3IYT42QTC63O7TMIZXIHT45KMF5HBBI6SHA, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.643408210 (60h59m47.36597125s) [noise=true , address=104.36.82.13:4186 , piece_id=BVJ63CB5CSNMYXBQDZKALAG4LB54K6ULIN6DRXXSJSGKKPBGFHMA, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.643711981 (60h59m47.366275021s) [noise=true , address=45.91.116.5:4299 , piece_id=QMXIG2BFBUCOEO7QXAMALMM5N7S4QOXUJGJJBLWS7M6M64SOCJXQ, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.644166210 (60h59m47.36672925s) [noise=true , address=45.91.116.12:4226 , piece_id=MZA4NR2C6MKWKHWYVYZWEORNFISSMO3AQZ3VY5MD2KIHHPTQZPLA, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.644226715 (60h59m47.366789755s) [noise=true , address=104.36.82.8:4972 , piece_id=LQTBSKXPO3ZYMWBW25T2LDZLYDJBX6O54LDQYAB3D52APCRDMVFQ, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.644452969 (60h59m47.367016009s) [noise=true , address=104.36.82.10:4079 , piece_id=QU4V6CVRPIVDNBFUKOBXO7UI4LGLLW56WKQMDC6MJTVHMBKUIXSQ, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:37.645456809 (60h59m47.368019849s) [noise=true , address=77.90.31.21:4878 , piece_id=M25XYSY4T27BAQO7IIUHS26SG3B4GWURGOABATIEV2IG2BE6GUCA, seg_pos={0 0}, upload=58486]: Uploading piece... Jul 27 00:35:39.390967743 (60h59m49.113530783s) [upload=58486]: Next returned nil segment Jul 27 00:35:39.390984314 (60h59m49.113547354s) [upload=58486]: Waiting for error group managing segments... Jul 27 00:35:39.666117093 (60h59m49.388680133s) [noise=true , address=45.91.116.13:4154 , piece_id=GZ7YGJ2L4DLFQIK7OF6B6K5QJVEEPMKLLG7SEMNC2MUO72FAXU5A, seg_pos={0 0}, upload=58486]: Done uploading piece. err: <nil> Jul 27 00:35:39.744981202 (60h59m49.467544242s) [noise=true , address=45.91.116.12:4226 , piece_id=MZA4NR2C6MKWKHWYVYZWEORNFISSMO3AQZ3VY5MD2KIHHPTQZPLA, seg_pos={0 0}, upload=58486]: Done uploading piece. err: <nil> Jul 27 00:35:39.748969784 (60h59m49.471532824s) [noise=true , address=77.90.31.16:4733 , piece_id=NMMZOBY3S6E63IIWMB7QDBKNFP5PM2RODTZPJRJXPR5IKLQAX4CA, seg_pos={0 0}, upload=58486]: Done uploading piece. err: <nil> Jul 27 00:35:39.763571331 (60h59m49.486134361s) [noise=true , address=45.91.116.5:4299 , piece_id=QMXIG2BFBUCOEO7QXAMALMM5N7S4QOXUJGJJBLWS7M6M64SOCJXQ, seg_pos={0 0}, upload=58486]: Done uploading piece. err: <nil> Jul 27 00:35:39.816383465 (60h59m49.538946505s) [noise=true , address=77.90.31.17:4771 , piece_id=EE7CMF5HQUKVCWFHKATGYXTQCSL3EDP4YA4TZ7BL4I4YR75JVG2A, seg_pos={0 0}, upload=58486]: Done uploading piece. err: <nil> Jul 27 00:36:06.766912240 (61h0m16.48947528s) [noise=true , address=104.36.82.10:4079 , piece_id=QU4V6CVRPIVDNBFUKOBXO7UI4LGLLW56WKQMDC6MJTVHMBKUIXSQ, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.766964259 (61h0m16.489527299s) [noise=true , address=104.36.82.14:4223 , piece_id=2DZEOIASHAVXTSJRCETE3R2MSYOCOZLWQJVXDBCDO2GFDPFALWVQ, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.766997954 (61h0m16.489560994s) [noise=true , address=104.36.82.15:4258 , piece_id=ISCHVFRTDGDJMQV3PWQCXFTS6ESLL7SN472HWSS6LMHPV4CGRTAQ, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:53816->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767029534 (61h0m16.489592574s) [noise=true , address=104.36.82.5:4838 , piece_id=PM3E3G2AYXXBPGAJYUO3YBCAC27VQPN76JVPHTPM4ALYAEVHKBJQ, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:36256->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767101532 (61h0m16.489664572s) [noise=true , address=77.90.31.15:4697 , piece_id=WH23YMYXHCGEE6OAU3IYT42QTC63O7TMIZXIHT45KMF5HBBI6SHA, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767264955 (61h0m16.489827995s) [noise=true , address=104.36.82.13:4186 , piece_id=BVJ63CB5CSNMYXBQDZKALAG4LB54K6ULIN6DRXXSJSGKKPBGFHMA, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:47726->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767304340 (61h0m16.48986738s) [noise=true , address=77.90.31.21:4878 , piece_id=M25XYSY4T27BAQO7IIUHS26SG3B4GWURGOABATIEV2IG2BE6GUCA, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767428618 (61h0m16.489991648s) [noise=true , address=104.36.82.11:4114 , piece_id=T75LXCDLLJH5AAA3DZ5RMNTSPITHR357J2XQ4VAGWJF2UESBLURA, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:57072->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767603131 (61h0m16.490166171s) [noise=true , address=104.36.82.8:4972 , piece_id=LQTBSKXPO3ZYMWBW25T2LDZLYDJBX6O54LDQYAB3D52APCRDMVFQ, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767635804 (61h0m16.490198844s) [noise=true , address=104.36.82.2:4730 , piece_id=CYXYHS3NHHZ4LAXL4SWV4N4JOGPXGA3TSFAFNYWDNEBWO37SJAAQ, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:55466->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767641095 (61h0m16.490204125s) [noise=true , address=77.90.31.18:4805 , piece_id=M6BDVBEP3S5OL3UBJXZSV54T4UHSTEWAZCVXM2ZDA3VLDZVN3RVQ, seg_pos={0 0}, upload=58486]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:35778->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:06.767663897 (61h0m16.490226948s) [seg_pos={0 0}, upload=58486]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 27 00:36:06.997501503 (61h0m16.720064543s) [upload=58486]: Done uploading segments... Jul 27 00:36:06.997521811 (61h0m16.720084851s) [upload=58486]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.232631095s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231071771s FAIL (2025-07-27 00:36:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== s) [noise=true , address=64.49.13.138:4482 , piece_id=EMPUJEPO2YKFXA55BXIGHHG5EXPA5L7HDXBI6IOGJUP47SHYTOEQ, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.424720863 (60h59m11.147283903s) [noise=true , address=77.90.31.19:4871 , piece_id=OGU2KTPMEDOHK6ZXB7MFFNMIS7JSJ4CS7GDQIPO6WC6E2EMDORJA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.425015086 (60h59m11.147578126s) [noise=true , address=45.91.116.10:4093 , piece_id=LMCMJUUSFJSTSTLIKIG6I26LEG6FABNFGO6K4NAZSJGS2CXZ5Q2A, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.425259433 (60h59m11.147822483s) [noise=true , address=77.90.31.18:4816 , piece_id=A4USNOIKDOGBDXFWTYU5KFYIAU52YXHNRCAUWCRDGFCEG2VQ26BA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.425511726 (60h59m11.148074766s) [noise=true , address=77.90.31.21:4885 , piece_id=ZMXGLQDLLIFUH554IN6HE2BYIE3HJYRBT2AIS2QPQJXE6AWVEVYQ, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.425759600 (60h59m11.14832265s) [noise=true , address=104.36.82.44:4872 , piece_id=ZJH4VGX4THRRMHDQ5VAZAFFEZHKIEKRKTMIT3WRIBCPRXUMGOQRQ, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.426025860 (60h59m11.14858891s) [noise=true , address=79.127.205.233:4639 , piece_id=NTJBINTDUPKDBCGYRWM4SQDXNJ3HMQMRIWC7RWRAQSO22H7KI2VA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.426257423 (60h59m11.148820473s) [noise=true , address=104.36.82.54:4186 , piece_id=TMKN6CXWCV3ID4CYJIAZW5BQDCTUZ6DW7MYFMLKNSIZO32MOS6UA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.426500688 (60h59m11.149063728s) [noise=true , address=104.36.82.46:4509 , piece_id=DFDNMNNGIHQSQNDZJKI37SGQS2N44XX6D3YJ4J7ZRR5NLXD3ENCQ, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.426743893 (60h59m11.149306934s) [noise=true , address=104.36.82.53:4133 , piece_id=6MMZJKPK3ASUPGLDFLHURVYLVTIDV3FXLVRCUXPY2B4FZLFP3UXA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.426981158 (60h59m11.149544198s) [noise=true , address=104.36.82.47:4531 , piece_id=QA4UHEHEVMSPVNAZWBP6FH3WMNS6SYWG2SYGMZU34WLEO2OHO3CA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.427223973 (60h59m11.149787013s) [noise=true , address=79.127.219.38:4005 , piece_id=TNUMFV6FO4IPC3KDURC2JREQE3HFRG4YG2Z5WV367764IPUZUYQA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.427467067 (60h59m11.150030107s) [noise=true , address=45.91.116.11:4130 , piece_id=YMZNWTM7A7RLDLGQKRTEYAGYUXRSXF4IKYM2KFWK2WBVEN2UOIHA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.427694713 (60h59m11.150257753s) [noise=true , address=79.127.205.246:4286 , piece_id=RAPXOGYXMAU3GEBN4KIGKRJ5B4WANPRSN4HLL65RBFIA3CD4VRSA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.428021959 (60h59m11.150584999s) [noise=true , address=45.91.116.4:4964 , piece_id=QUPIZM74NSYNX3GXRORGOXARKFKANOZE2NIL2CXQ2X65RXKOILUQ, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:01.428358022 (60h59m11.150921072s) [noise=true , address=79.127.205.245:4248 , piece_id=7TVY3IKBRZYW72EPNXUFMZE2ASY7KTIRPD65TKKGZMXVMVIC2DFA, seg_pos={0 0}, upload=58478]: Uploading piece... Jul 27 00:35:02.336494689 (60h59m12.059057729s) [noise=true , address=79.127.220.101:4041 , piece_id=I7QB6IMUV577TCKKELUJL7EQDOT2FWUGRBG5QM3ZNXF6UWOBXBZA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:35:02.538227141 (60h59m12.260790181s) [upload=58478]: Next returned nil segment Jul 27 00:35:02.538246398 (60h59m12.260809438s) [upload=58478]: Waiting for error group managing segments... Jul 27 00:35:02.846805700 (60h59m12.56936874s) [noise=true , address=79.127.205.233:4639 , piece_id=NTJBINTDUPKDBCGYRWM4SQDXNJ3HMQMRIWC7RWRAQSO22H7KI2VA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: <nil> Jul 27 00:35:02.847973495 (60h59m12.570536535s) [noise=true , address=79.127.205.246:4286 , piece_id=RAPXOGYXMAU3GEBN4KIGKRJ5B4WANPRSN4HLL65RBFIA3CD4VRSA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: <nil> Jul 27 00:35:02.877681875 (60h59m12.600244916s) [noise=true , address=79.127.205.245:4248 , piece_id=7TVY3IKBRZYW72EPNXUFMZE2ASY7KTIRPD65TKKGZMXVMVIC2DFA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: <nil> Jul 27 00:35:03.246352777 (60h59m12.968915817s) [noise=true , address=45.91.116.10:4093 , piece_id=LMCMJUUSFJSTSTLIKIG6I26LEG6FABNFGO6K4NAZSJGS2CXZ5Q2A, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:35:03.443035047 (60h59m13.165598087s) [noise=true , address=79.127.219.38:4005 , piece_id=TNUMFV6FO4IPC3KDURC2JREQE3HFRG4YG2Z5WV367764IPUZUYQA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: <nil> Jul 27 00:35:03.444610180 (60h59m13.16717323s) [noise=true , address=45.91.116.11:4130 , piece_id=YMZNWTM7A7RLDLGQKRTEYAGYUXRSXF4IKYM2KFWK2WBVEN2UOIHA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: <nil> Jul 27 00:35:03.915695421 (60h59m13.638258471s) [noise=true , address=45.91.116.4:4964 , piece_id=QUPIZM74NSYNX3GXRORGOXARKFKANOZE2NIL2CXQ2X65RXKOILUQ, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:35:07.736421161 (60h59m17.458984191s) [noise=true , address=64.49.13.138:4482 , piece_id=EMPUJEPO2YKFXA55BXIGHHG5EXPA5L7HDXBI6IOGJUP47SHYTOEQ, seg_pos={0 0}, upload=58478]: Done uploading piece. err: <nil> Jul 27 00:35:08.249425374 (60h59m17.971988414s) [noise=true , address=77.90.31.19:4871 , piece_id=OGU2KTPMEDOHK6ZXB7MFFNMIS7JSJ4CS7GDQIPO6WC6E2EMDORJA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: <nil> Jul 27 00:36:00.337143031 (61h0m10.059706071s) [noise=true , address=104.36.82.47:4531 , piece_id=QA4UHEHEVMSPVNAZWBP6FH3WMNS6SYWG2SYGMZU34WLEO2OHO3CA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op Jul 27 00:36:00.337531124 (61h0m10.060094154s) [noise=true , address=104.36.82.44:4872 , piece_id=ZJH4VGX4THRRMHDQ5VAZAFFEZHKIEKRKTMIT3WRIBCPRXUMGOQRQ, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:104.36.82.44:4872): protocol: rpc: dial tcp 104.36.82.44:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:00.337588253 (61h0m10.060151293s) [noise=true , address=77.90.31.18:4816 , piece_id=A4USNOIKDOGBDXFWTYU5KFYIAU52YXHNRCAUWCRDGFCEG2VQ26BA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: rpc: dial tcp 77.90.31.18:4816: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:00.337623561 (61h0m10.060186591s) [noise=true , address=104.36.82.46:4509 , piece_id=DFDNMNNGIHQSQNDZJKI37SGQS2N44XX6D3YJ4J7ZRR5NLXD3ENCQ, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: write tcp 188.166.225.170:60248->104.36.82.46:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:00.337710968 (61h0m10.060274008s) [noise=true , address=104.36.82.53:4133 , piece_id=6MMZJKPK3ASUPGLDFLHURVYLVTIDV3FXLVRCUXPY2B4FZLFP3UXA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: rpc: dial tcp 104.36.82.53:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:00.338007115 (61h0m10.060570155s) [noise=true , address=104.36.82.52:4112 , piece_id=PBI3RF6AUUJL7U2RDBXAYKUQU32DBR5PQORQU6CDW7QVGIP7A5IQ, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: rpc: dial tcp 104.36.82.52:4112: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:00.338050918 (61h0m10.060613968s) [noise=true , address=104.36.82.54:4186 , piece_id=TMKN6CXWCV3ID4CYJIAZW5BQDCTUZ6DW7MYFMLKNSIZO32MOS6UA, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: rpc: dial tcp 104.36.82.54:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:00.338089122 (61h0m10.060652172s) [noise=true , address=77.90.31.21:4885 , piece_id=ZMXGLQDLLIFUH554IN6HE2BYIE3HJYRBT2AIS2QPQJXE6AWVEVYQ, seg_pos={0 0}, upload=58478]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:00.338104080 (61h0m10.06066712s) [seg_pos={0 0}, upload=58478]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:36:00.566058593 (61h0m10.288621633s) [upload=58478]: Done uploading segments... Jul 27 00:36:00.566084563 (61h0m10.288647603s) [upload=58478]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.231089244s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [...XX..XX.X.XX...X...............XX......X...XXX..XXX..X..X.] duration: 14.521628325s FAIL (2025-07-27 00:35:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.521643926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X..........................................X.............] duration: 1.833681063s FAIL (2025-07-27 00:35:40 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.83371004s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XX] duration: 30.237724188s FAIL (2025-07-27 00:35:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== lose: conn closed: op Jul 27 00:34:27.903947314 (60h58m37.626510354s) [seg_pos={0 0}, upload=58464]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:34:28.134454290 (60h58m37.85701738s) [upload=58464]: Done uploading segments... Jul 27 00:34:28.134480620 (60h58m37.85704366s) [upload=58464]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:35:06.924949102 (60h59m16.647512142s) [upload=58481]: Starting upload Jul 27 00:35:06.925004648 (60h59m16.647567689s) [upload=58481]: Done starting upload Jul 27 00:35:06.925097726 (60h59m16.647660766s) [upload=58481]: Uploading segments... Jul 27 00:35:06.925285966 (60h59m16.647849007s) [upload=58481]: Got next segment. Inline: false Jul 27 00:35:06.925320112 (60h59m16.647883152s) [upload=58481]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:35:06.925325302 (60h59m16.647888342s) [upload=58481]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:35:07.156457765 (60h59m16.879020795s) [seg_pos={0 0}, upload=58481]: Begin upload segment... Jul 27 00:35:07.156501679 (60h59m16.87906472s) [seg_pos={0 0}, upload=58481]: Done begin upload segment. Jul 27 00:35:07.156524102 (60h59m16.879087142s) [noise=true , address=45.91.116.7:4010 , piece_id=UDKND5MDDG5SP5SQ55G5JUI4GWMSQRF6BETMAUQ5XOPAKHTE6HKQ, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.156923226 (60h59m16.879486256s) [noise=true , address=45.91.116.11:4119 , piece_id=QI7EIWMNJWKXMOOL5VH5C54OB5RBM7LMJ4TFA4M7QA526LFTHV6A, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.157205195 (60h59m16.879768245s) [noise=true , address=77.90.31.21:4878 , piece_id=2MPVFJEQHAXJMDQLK7VM7FREQGLODVCWRALHVCLLRPWKDJR2ZDFQ, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.157467367 (60h59m16.880030407s) [noise=true , address=77.90.31.13:4628 , piece_id=2V454BARJNGC4VUULDSQFKEVUV32EYAENYORBQQCEFOJDYQLYIBA, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.158331029 (60h59m16.880894059s) [noise=true , address=104.36.82.4:4802 , piece_id=CGMHDCVZNEKGT4Y4KXNN3H2DSZV62HNQYLLU5NAR7JG25WKYATKQ, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.158348503 (60h59m16.880911543s) [noise=true , address=77.90.31.11:4555 , piece_id=QILFOJNOQL353ZMBDAQZ4BEX3U4B3LPJY5LJ7Z6YVO6NDLIND6IA, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.158586117 (60h59m16.881149157s) [noise=true , address=45.91.116.10:4083 , piece_id=Z2SB5X65YI2ERSMKBFLVQPUZ5LR7EL5RL73KMEG73QID72S3LULA, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.158658736 (60h59m16.881221766s) [noise=true , address=104.36.82.8:4972 , piece_id=DB4SY4JTVGF52U7TXB3XXGIGMJ6TU3AY7S4SWBQLUSA6MYP6B3IA, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.158791520 (60h59m16.88135456s) [noise=true , address=45.91.116.8:4045 , piece_id=HU6USPQ2IJ4PG75FNZ56YZE3QWA3M5EDNAJ773EOC4HE2W2BC67Q, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.158945985 (60h59m16.881509025s) [noise=true , address=104.36.82.3:4765 , piece_id=IS2MFV4KJQUOD677ZDZY3H6RO4IGZTY7QKNIHTMSXQTDTDN7OP6Q, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.159037631 (60h59m16.881600671s) [noise=true , address=77.90.31.19:4843 , piece_id=CTMFOKET3Y5ZRCPF75MT353R3ZHILBGBZSFW6SXKJX3WF7AOZU3Q, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.159215531 (60h59m16.881778571s) [noise=true , address=45.91.116.4:4937 , piece_id=K5ITSKETYXYCM3XMFNSAVMP7Y5GPZO6F5MCJAZBR5ZPD5SVJSUUQ, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.159424241 (60h59m16.881987281s) [noise=true , address=77.90.31.15:4697 , piece_id=EMGJRR5T6JJM6B3XSDHMCUGCAMGY44TMTVE4376D7CXDJPA7LLCQ, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.159459849 (60h59m16.882022879s) [noise=true , address=77.90.31.14:4661 , piece_id=S5QRAGEH2MNTYDHZXPAHU5UEL754ONWYQICAWV54364SGGULDOMQ, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.159663318 (60h59m16.882226358s) [noise=true , address=45.91.116.12:4225 , piece_id=7ERUJ3PIRUV3MNZ2QMP3VCWFOWT6R5NBGXFHOGGQ4BUIBGQ4EAPA, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:07.159989392 (60h59m16.882552432s) [noise=true , address=104.36.82.5:4840 , piece_id=OWDKUWF4SKEOKUKR6M5I3ESGEJGMUCCYNISCBJWGAP6RVSOGQPOQ, seg_pos={0 0}, upload=58481]: Uploading piece... Jul 27 00:35:08.880310158 (60h59m18.602873198s) [upload=58481]: Next returned nil segment Jul 27 00:35:08.880331057 (60h59m18.602894097s) [upload=58481]: Waiting for error group managing segments... Jul 27 00:35:09.254207310 (60h59m18.97677034s) [noise=true , address=45.91.116.12:4225 , piece_id=7ERUJ3PIRUV3MNZ2QMP3VCWFOWT6R5NBGXFHOGGQ4BUIBGQ4EAPA, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:09.256702393 (60h59m18.979265443s) [noise=true , address=77.90.31.14:4661 , piece_id=S5QRAGEH2MNTYDHZXPAHU5UEL754ONWYQICAWV54364SGGULDOMQ, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:09.272939660 (60h59m18.9955027s) [noise=true , address=45.91.116.7:4010 , piece_id=UDKND5MDDG5SP5SQ55G5JUI4GWMSQRF6BETMAUQ5XOPAKHTE6HKQ, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:09.279299869 (60h59m19.001862909s) [noise=true , address=45.91.116.10:4083 , piece_id=Z2SB5X65YI2ERSMKBFLVQPUZ5LR7EL5RL73KMEG73QID72S3LULA, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:09.303980870 (60h59m19.026543901s) [noise=true , address=45.91.116.11:4119 , piece_id=QI7EIWMNJWKXMOOL5VH5C54OB5RBM7LMJ4TFA4M7QA526LFTHV6A, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:09.322745923 (60h59m19.045308963s) [noise=true , address=77.90.31.11:4555 , piece_id=QILFOJNOQL353ZMBDAQZ4BEX3U4B3LPJY5LJ7Z6YVO6NDLIND6IA, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:09.323828013 (60h59m19.046391053s) [noise=true , address=45.91.116.8:4045 , piece_id=HU6USPQ2IJ4PG75FNZ56YZE3QWA3M5EDNAJ773EOC4HE2W2BC67Q, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:09.420553728 (60h59m19.143116758s) [noise=true , address=45.91.116.4:4937 , piece_id=K5ITSKETYXYCM3XMFNSAVMP7Y5GPZO6F5MCJAZBR5ZPD5SVJSUUQ, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:10.999558043 (60h59m20.722121083s) [noise=true , address=77.90.31.13:4628 , piece_id=2V454BARJNGC4VUULDSQFKEVUV32EYAENYORBQQCEFOJDYQLYIBA, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:11.174054695 (60h59m20.896617735s) [noise=true , address=77.90.31.19:4843 , piece_id=CTMFOKET3Y5ZRCPF75MT353R3ZHILBGBZSFW6SXKJX3WF7AOZU3Q, seg_pos={0 0}, upload=58481]: Done uploading piece. err: <nil> Jul 27 00:35:36.279442936 (60h59m46.002005976s) [noise=true , address=104.36.82.3:4765 , piece_id=IS2MFV4KJQUOD677ZDZY3H6RO4IGZTY7QKNIHTMSXQTDTDN7OP6Q, seg_pos={0 0}, upload=58481]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:38178->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:36.279507299 (60h59m46.002070339s) [noise=true , address=104.36.82.5:4840 , piece_id=OWDKUWF4SKEOKUKR6M5I3ESGEJGMUCCYNISCBJWGAP6RVSOGQPOQ, seg_pos={0 0}, upload=58481]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:36408->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:36.279515856 (60h59m46.002078886s) [noise=true , address=104.36.82.4:4802 , piece_id=CGMHDCVZNEKGT4Y4KXNN3H2DSZV62HNQYLLU5NAR7JG25WKYATKQ, seg_pos={0 0}, upload=58481]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:36.279539110 (60h59m46.00210215s) [noise=true , address=77.90.31.15:4697 , piece_id=EMGJRR5T6JJM6B3XSDHMCUGCAMGY44TMTVE4376D7CXDJPA7LLCQ, seg_pos={0 0}, upload=58481]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:36.279555231 (60h59m46.002118271s) [noise=true , address=77.90.31.21:4878 , piece_id=2MPVFJEQHAXJMDQLK7VM7FREQGLODVCWRALHVCLLRPWKDJR2ZDFQ, seg_pos={0 0}, upload=58481]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:36.279649331 (60h59m46.002212361s) [noise=true , address=104.36.82.8:4972 , piece_id=DB4SY4JTVGF52U7TXB3XXGIGMJ6TU3AY7S4SWBQLUSA6MYP6B3IA, seg_pos={0 0}, upload=58481]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:59520->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:36.279680170 (60h59m46.00224321s) [seg_pos={0 0}, upload=58481]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:35:36.515614992 (60h59m46.238178032s) [upload=58481]: Done uploading segments... Jul 27 00:35:36.515638767 (60h59m46.238201807s) [upload=58481]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.237760569s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X...XX..XX.X.XX...X...............XX......X...XXX..XXX..X..X] duration: 1m0.00048714s FAIL (2025-07-27 00:35:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000497078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.] duration: 7.159398484s FAIL (2025-07-27 00:35:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.159419695s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X....XXX.X...X......X.XX...XXXX.....X..........X..X..X...X] duration: 30.000370711s FAIL (2025-07-27 00:35:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000391271s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234493291s FAIL (2025-07-27 00:35:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .947246070 (60h58m10.66980911s) [noise=true , address=79.127.205.234:4606 , piece_id=BPO2QLZLYDMLDFG75PZWRQEDLFVYHATXPOSFEHNOW5TTQVHJNCDQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.947411467 (60h58m10.669974507s) [noise=true , address=79.127.201.214:4321 , piece_id=QNIQHFLYGDYMJAKX34J25OWNOSBJ6ARDNOQN6ORKKJJ5YCPN5BIQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.947611719 (60h58m10.67017476s) [noise=true , address=104.36.82.56:4243 , piece_id=NV4M2T6H2JQ3BNLQEWTDIFWQSRASLNC6XAMUTTSROGZAOUBLH2FQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.947730897 (60h58m10.670293937s) [noise=true , address=104.36.82.55:4211 , piece_id=ZX3TNIL6OVFU7CL6HQWWBB2D2OZG3QQF2PCA4FF2CRD2TPD5PURQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948008538 (60h58m10.670571578s) [noise=true , address=79.127.201.215:4349 , piece_id=5JW3GKNWRETNYWAMWDTDCYG4STMUG5AJTJARZS2EDCMUETXWLUAQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948329142 (60h58m10.670892182s) [noise=true , address=104.36.82.47:4517 , piece_id=3SPHKVUJIB4LTR32O6P6DOHKD2LZZ3BZS42GDJAH6MML4P2JH4QA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948347508 (60h58m10.670910548s) [noise=true , address=104.36.82.4:4828 , piece_id=TGD6QOF3TQDNNFMS2O3V752GGMUFJYHVO63MINLF4MPFVSE6D46A, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948617704 (60h58m10.671180744s) [noise=true , address=45.91.116.10:4093 , piece_id=SBUAEHFRMRG5MF3K7CIT54MOGMINSBDMMAPMUCAN7MICW4HNZQHA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948722424 (60h58m10.671285464s) [noise=true , address=79.127.205.235:4688 , piece_id=YJHAWDUPMGXZLOXM443YGCBFAC6VC5PQTODFIPSGOSR34UEUXJVQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.948856280 (60h58m10.671419321s) [noise=true , address=104.36.82.54:4165 , piece_id=RQUNZJHXQZX2QSWBPMFNGWSZGWGBT4PQHA2BOHHIKRSX4OKUQK6A, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949040964 (60h58m10.671603994s) [noise=true , address=79.127.219.38:4001 , piece_id=BK54RX4RCXMUMPTVO3SNEMCUDT3Q7VHPLEU2QPSYV2R4QJJAEMRA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949175291 (60h58m10.671738331s) [noise=true , address=104.36.82.49:4978 , piece_id=U5TL3JMP3WMY6WXQAIIXWW3NWTRMUUSYCEUM5EMJRK67PELHGSBQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949339656 (60h58m10.671902696s) [noise=true , address=156.146.43.225:4449 , piece_id=LVQQLBP5XGRQJLIWFJH6L2HJHALCI4KP5XQDURUJJUUUYVVZDFEA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949404449 (60h58m10.671967489s) [noise=true , address=77.90.31.10:4530 , piece_id=VXE6EFJ2FLEEDNJAFFQIN3RGM5UBJXR6YER7CEN65RDPUYF53CCQ, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.949647995 (60h58m10.672211036s) [noise=true , address=79.127.205.246:4288 , piece_id=47LEBRSACX5JQM426GON2AE6TMOPGP2TMGQFEXFKU5NJONHSZMAA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:00.954106295 (60h58m10.676669335s) [noise=true , address=104.36.82.52:4085 , piece_id=7MLVFJ64SLANN6PHAS2BQQLCYJ3JNPIRFDXGJU4ONGPEVC3FPRMA, seg_pos={0 0}, upload=58466]: Uploading piece... Jul 27 00:34:02.075052567 (60h58m11.797615607s) [upload=58466]: Next returned nil segment Jul 27 00:34:02.075075701 (60h58m11.797638741s) [upload=58466]: Waiting for error group managing segments... Jul 27 00:34:02.374234097 (60h58m12.096797137s) [noise=true , address=79.127.205.234:4606 , piece_id=BPO2QLZLYDMLDFG75PZWRQEDLFVYHATXPOSFEHNOW5TTQVHJNCDQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.375144979 (60h58m12.097708029s) [noise=true , address=79.127.205.246:4288 , piece_id=47LEBRSACX5JQM426GON2AE6TMOPGP2TMGQFEXFKU5NJONHSZMAA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.386272419 (60h58m12.108835459s) [noise=true , address=79.127.205.235:4688 , piece_id=YJHAWDUPMGXZLOXM443YGCBFAC6VC5PQTODFIPSGOSR34UEUXJVQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.865938136 (60h58m12.588501177s) [noise=true , address=156.146.43.225:4449 , piece_id=LVQQLBP5XGRQJLIWFJH6L2HJHALCI4KP5XQDURUJJUUUYVVZDFEA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.867764060 (60h58m12.5903271s) [noise=true , address=79.127.219.38:4001 , piece_id=BK54RX4RCXMUMPTVO3SNEMCUDT3Q7VHPLEU2QPSYV2R4QJJAEMRA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:02.994582891 (60h58m12.717145921s) [noise=true , address=79.127.201.214:4321 , piece_id=QNIQHFLYGDYMJAKX34J25OWNOSBJ6ARDNOQN6ORKKJJ5YCPN5BIQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:03.382397019 (60h58m13.104960059s) [noise=true , address=64.49.13.26:4953 , piece_id=ZXNFKT5IJ5MIXNAKTOX3GL6PG3AROWLQWUSIYX42HF7XRJM2FR3A, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:34:04.630471883 (60h58m14.353034923s) [noise=true , address=45.91.116.10:4093 , piece_id=SBUAEHFRMRG5MF3K7CIT54MOGMINSBDMMAPMUCAN7MICW4HNZQHA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:34:04.787587931 (60h58m14.510150971s) [noise=true , address=77.90.31.10:4530 , piece_id=VXE6EFJ2FLEEDNJAFFQIN3RGM5UBJXR6YER7CEN65RDPUYF53CCQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:29.756831010 (60h58m39.47939405s) [noise=true , address=79.127.201.215:4349 , piece_id=5JW3GKNWRETNYWAMWDTDCYG4STMUG5AJTJARZS2EDCMUETXWLUAQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: <nil> Jul 27 00:34:59.866440719 (60h59m9.589003759s) [noise=true , address=104.36.82.21:4489 , piece_id=GK7F7VIIDKRHGNO4ZTKRRUYSRYXU7L5PSYKGYFPIYKCXOFLR76KA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12fao48Sd4XTApMqhudk7rT4pk6FssZRoqNdoUnrkL2b8cyExc5, address:104.36.82.21:4489): protocol: rpc: dial tcp 104.36.82.21:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866482488 (60h59m9.589045528s) [noise=true , address=104.36.82.55:4211 , piece_id=ZX3TNIL6OVFU7CL6HQWWBB2D2OZG3QQF2PCA4FF2CRD2TPD5PURQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: rpc: dial tcp 104.36.82.55:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866510051 (60h59m9.589073091s) [noise=true , address=104.36.82.47:4517 , piece_id=3SPHKVUJIB4LTR32O6P6DOHKD2LZZ3BZS42GDJAH6MML4P2JH4QA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866547573 (60h59m9.589110613s) [noise=true , address=104.36.82.56:4243 , piece_id=NV4M2T6H2JQ3BNLQEWTDIFWQSRASLNC6XAMUTTSROGZAOUBLH2FQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: rpc: dial tcp 104.36.82.56:4243: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866564525 (60h59m9.589127565s) [noise=true , address=104.36.82.4:4828 , piece_id=TGD6QOF3TQDNNFMS2O3V752GGMUFJYHVO63MINLF4MPFVSE6D46A, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:42892->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866622546 (60h59m9.589185606s) [noise=true , address=104.36.82.54:4165 , piece_id=RQUNZJHXQZX2QSWBPMFNGWSZGWGBT4PQHA2BOHHIKRSX4OKUQK6A, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866627927 (60h59m9.589190967s) [noise=true , address=104.36.82.49:4978 , piece_id=U5TL3JMP3WMY6WXQAIIXWW3NWTRMUUSYCEUM5EMJRK67PELHGSBQ, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:104.36.82.49:4978): protocol: write tcp 188.166.225.170:53440->104.36.82.49:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866663966 (60h59m9.589226996s) [noise=true , address=104.36.82.52:4085 , piece_id=7MLVFJ64SLANN6PHAS2BQQLCYJ3JNPIRFDXGJU4ONGPEVC3FPRMA, seg_pos={0 0}, upload=58466]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:37302->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:59.866688292 (60h59m9.589251332s) [seg_pos={0 0}, upload=58466]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:35:00.098519222 (60h59m9.821082262s) [upload=58466]: Done uploading segments... Jul 27 00:35:00.098551884 (60h59m9.821114924s) [upload=58466]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.234516887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X..........................................X............] duration: 2.045918278s FAIL (2025-07-27 00:34:49 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.045954647s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX] duration: 30.236702967s FAIL (2025-07-27 00:34:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.236726912s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)