Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233719931s FAIL (2025-07-26 23:23:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== noise=true , address=79.127.205.245:4252 , piece_id=ETI2L26HQI6FFTZVFRDRASPFVYPV73S7ZP6OQ6PQ5E6PT43DPBTA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.457179577 (59h47m0.179742617s) [noise=true , address=79.127.205.234:4609 , piece_id=A4P53OX6JWPZUPAWAYJH5RZVKNDUSWPCNGZ4DE5SRBLVBUJYUSLA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.457611865 (59h47m0.180174905s) [noise=true , address=79.127.205.233:4644 , piece_id=RH4Q4IUGPNAYALV2IMCT3JV7PA6VEQEO6UVHCQUEZ67KQ6U3SCXA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.457825744 (59h47m0.180388784s) [noise=true , address=77.90.31.19:4850 , piece_id=5W6S6WGGSYXNFP7RDVWFAHYHS7EVL3EE6PWAJY5ZOJKY2R3ZGKKA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458023182 (59h47m0.180586222s) [noise=true , address=77.90.31.10:4543 , piece_id=OQGZGJHBI4XLMDCWOTTYYRCIYMERYYFTCWP4R3HHAOW2E5QM3ZZQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458130166 (59h47m0.180693206s) [noise=true , address=104.36.82.46:4501 , piece_id=CMQL7Z633XDFHTUORI4WAKZSKQKFP3NP426JJN5VTMMTPZC6JAFQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458309749 (59h47m0.18087279s) [noise=true , address=104.36.82.54:4168 , piece_id=BQ35DZDJRP3BYGXWTNB6OX6BSYJYSEH3OA3NEII32S7SVMFGN6AQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458559056 (59h47m0.181122096s) [noise=true , address=79.127.205.235:4666 , piece_id=JYBLHKDXS335W6XJ44Q3INQLL5JGLAVIPY2V4KOTMMOLOV3RS5TQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458600296 (59h47m0.181163336s) [noise=true , address=104.36.82.47:4538 , piece_id=33ZLU5ZWCTOMWTSM6XVXKG5JYFX3M4VMXPYXTREUANSLLOD3UM5Q, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.458975032 (59h47m0.181538072s) [noise=true , address=104.36.82.56:4252 , piece_id=W3ST6N4BXGTG4WYMQD2QSRDXVZUDEE7DSBZJVF7ME7E43WIWEBHA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.459217476 (59h47m0.181780516s) [noise=true , address=77.90.31.16:4761 , piece_id=MRRN4A7XRBOSQBPDJ6ZDXJWQZYG2OP7JSLGCTUDVAQXARLIFSY2A, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.459369087 (59h47m0.181932127s) [noise=true , address=77.90.31.18:4814 , piece_id=NNTX227RQG7SAXZDK6SS4FGVKVWQAD5NBU6RA4DZEHIMN3HZKBSQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.459702925 (59h47m0.182265955s) [noise=true , address=45.91.116.12:4230 , piece_id=RL62YWLYNVJENMCOSXD4N47CJ46QROXWGBNEAD2RP27ZDO2PL7WQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.460005374 (59h47m0.182568414s) [noise=true , address=104.36.82.55:4225 , piece_id=IA7W7XF7ACC35CVLPOIATVVS3FW3MCCHCSTHGKM4ZV6FN5CG5FTQ, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.460516111 (59h47m0.183079151s) [noise=true , address=45.91.116.14:4222 , piece_id=72HI7P3FRZICTSKYWQU6HFS64Z6KL6TY3IBUXLRYSAOQWCCIX44A, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.460829801 (59h47m0.183392841s) [noise=true , address=77.90.31.11:4580 , piece_id=X6H5XSP4TDEFHDJVVVQZE2CAXAKWGPAFKTXBDSWACAYB52GERIOA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.461137901 (59h47m0.183700941s) [noise=true , address=104.36.82.52:4101 , piece_id=FE2KQOZ5Q2SHMDHIVGYN2KNTUD7CPSYXCC73JHVARTTW56DGPP2A, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:50.461545511 (59h47m0.184108541s) [noise=true , address=45.91.116.9:4270 , piece_id=VQLSZYUJ35BRDMCYJ2WP3UW2JRLMZSJ4XVJIJILW4BYB2T6U6KRA, seg_pos={0 0}, upload=57356]: Uploading piece... Jul 26 23:22:51.571288125 (59h47m1.293851165s) [upload=57356]: Next returned nil segment Jul 26 23:22:51.571307603 (59h47m1.293870653s) [upload=57356]: Waiting for error group managing segments... Jul 26 23:22:51.851281448 (59h47m1.573844478s) [noise=true , address=79.127.205.233:4644 , piece_id=RH4Q4IUGPNAYALV2IMCT3JV7PA6VEQEO6UVHCQUEZ67KQ6U3SCXA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:51.864079465 (59h47m1.586642505s) [noise=true , address=79.127.205.234:4609 , piece_id=A4P53OX6JWPZUPAWAYJH5RZVKNDUSWPCNGZ4DE5SRBLVBUJYUSLA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:51.868213846 (59h47m1.590776886s) [noise=true , address=79.127.205.235:4666 , piece_id=JYBLHKDXS335W6XJ44Q3INQLL5JGLAVIPY2V4KOTMMOLOV3RS5TQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:51.882554543 (59h47m1.605117583s) [noise=true , address=79.127.205.245:4252 , piece_id=ETI2L26HQI6FFTZVFRDRASPFVYPV73S7ZP6OQ6PQ5E6PT43DPBTA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:52.331202453 (59h47m2.053765493s) [noise=true , address=45.91.116.12:4230 , piece_id=RL62YWLYNVJENMCOSXD4N47CJ46QROXWGBNEAD2RP27ZDO2PL7WQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:52.467399199 (59h47m2.189962239s) [noise=true , address=77.90.31.11:4580 , piece_id=X6H5XSP4TDEFHDJVVVQZE2CAXAKWGPAFKTXBDSWACAYB52GERIOA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:52.497447411 (59h47m2.220010441s) [noise=true , address=77.90.31.18:4814 , piece_id=NNTX227RQG7SAXZDK6SS4FGVKVWQAD5NBU6RA4DZEHIMN3HZKBSQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:52.889160627 (59h47m2.611723667s) [noise=true , address=45.91.116.9:4270 , piece_id=VQLSZYUJ35BRDMCYJ2WP3UW2JRLMZSJ4XVJIJILW4BYB2T6U6KRA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:22:53.886874268 (59h47m3.609437308s) [noise=true , address=45.91.116.14:4222 , piece_id=72HI7P3FRZICTSKYWQU6HFS64Z6KL6TY3IBUXLRYSAOQWCCIX44A, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:23:12.575738489 (59h47m22.298301529s) [noise=true , address=77.90.31.19:4850 , piece_id=5W6S6WGGSYXNFP7RDVWFAHYHS7EVL3EE6PWAJY5ZOJKY2R3ZGKKA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: <nil> Jul 26 23:23:49.324977343 (59h47m59.047540383s) [noise=true , address=104.36.82.46:4501 , piece_id=CMQL7Z633XDFHTUORI4WAKZSKQKFP3NP426JJN5VTMMTPZC6JAFQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: rpc: dial tcp 104.36.82.46:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325025095 (59h47m59.047588125s) [noise=true , address=104.36.82.56:4252 , piece_id=W3ST6N4BXGTG4WYMQD2QSRDXVZUDEE7DSBZJVF7ME7E43WIWEBHA, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:1UDFNiqXb4riRuewJDPrKFykVWLiryNKqgkH6uCZszz6Sr8naB, address:104.36.82.56:4252): protocol: rpc: dial tcp 104.36.82.56:4252: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325057617 (59h47m59.047620657s) [noise=true , address=77.90.31.16:4761 , piece_id=MRRN4A7XRBOSQBPDJ6ZDXJWQZYG2OP7JSLGCTUDVAQXARLIFSY2A, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 188.166.225.170:37552->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325073617 (59h47m59.047636647s) [noise=true , address=77.90.31.10:4543 , piece_id=OQGZGJHBI4XLMDCWOTTYYRCIYMERYYFTCWP4R3HHAOW2E5QM3ZZQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325096481 (59h47m59.047659521s) [noise=true , address=104.36.82.47:4538 , piece_id=33ZLU5ZWCTOMWTSM6XVXKG5JYFX3M4VMXPYXTREUANSLLOD3UM5Q, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325115728 (59h47m59.047678768s) [noise=true , address=104.36.82.52:4101 , piece_id=FE2KQOZ5Q2SHMDHIVGYN2KNTUD7CPSYXCC73JHVARTTW56DGPP2A, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 188.166.225.170:34934->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325126338 (59h47m59.047689378s) [noise=true , address=104.36.82.55:4225 , piece_id=IA7W7XF7ACC35CVLPOIATVVS3FW3MCCHCSTHGKM4ZV6FN5CG5FTQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: rpc: dial tcp 104.36.82.55:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325131569 (59h47m59.047694599s) [noise=true , address=104.36.82.54:4168 , piece_id=BQ35DZDJRP3BYGXWTNB6OX6BSYJYSEH3OA3NEII32S7SVMFGN6AQ, seg_pos={0 0}, upload=57356]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: rpc: dial tcp 104.36.82.54:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:49.325161436 (59h47m59.047724476s) [seg_pos={0 0}, upload=57356]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:23:49.556768540 (59h47m59.27933158s) [upload=57356]: Done uploading segments... Jul 26 23:23:49.556797816 (59h47m59.279360856s) [upload=57356]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.233731383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X........X.X...........................XXXXXXXXXXXXXXXXXXXX] duration: 30.231713323s FAIL (2025-07-26 23:23:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.231725266s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.......X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234611787s FAIL (2025-07-26 23:23:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== ing upload Jul 26 23:22:57.001061404 (59h47m6.723624514s) [upload=57357]: Uploading segments... Jul 26 23:22:57.001349515 (59h47m6.723912585s) [upload=57357]: Got next segment. Inline: false Jul 26 23:22:57.001397407 (59h47m6.723960487s) [upload=57357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:22:57.001405772 (59h47m6.723968852s) [upload=57357]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:57.239992641 (59h47m6.962555671s) [seg_pos={0 0}, upload=57357]: Begin upload segment... Jul 26 23:22:57.240036305 (59h47m6.962599345s) [seg_pos={0 0}, upload=57357]: Done begin upload segment. Jul 26 23:22:57.240057246 (59h47m6.962620276s) [noise=true , address=104.36.82.3:4765 , piece_id=ISSO3GMPZOCINXDFUC35BZD3SSHLEYIKG2WFLSTETRBHFWQARHFA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240114756 (59h47m6.962677796s) [noise=true , address=45.91.116.7:4011 , piece_id=NAORINYA6WT5JNHQIOXP6I5SBG4CRPPRDRBTVQVOVE6XZPFJM7FA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240465437 (59h47m6.963028467s) [noise=true , address=104.36.82.2:4729 , piece_id=6GB5QRQ3R7PF5KMWJO4IOYP32WEGDI6KKDG35IYQQXP4DFKHJCKQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240731295 (59h47m6.963294335s) [noise=true , address=45.91.116.10:4083 , piece_id=MIMPJTD7WHDGTW4MS4XP3HY3SGLGKWF4OICZBU64U5M63P4DC2NQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240769839 (59h47m6.963332879s) [noise=true , address=104.36.82.10:4078 , piece_id=TRH6SMMVIMOEHYAMEWSOUUCLRD3YDWZ7AUUTWRJSXUDR3WU3M6AA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.240995010 (59h47m6.96355805s) [noise=true , address=104.36.82.6:4873 , piece_id=3TYPHXHEXZAJFTNZZO2Z7FHA5LTYYQGGDOCGNRLK4RJULLLCJGMQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241130880 (59h47m6.96369392s) [noise=true , address=77.90.31.15:4697 , piece_id=TNRCHE4MVDYP2K4PCXGFXJ4BT7TQLGOO3DQBRUY6DYMFBQK3K75A, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241254456 (59h47m6.963817496s) [noise=true , address=77.90.31.18:4805 , piece_id=AP2IJBH26NFU7JQITMNVOQTQKBWEGMPB2FCIB4RJJAFNIVSRHZYQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241482341 (59h47m6.964045382s) [noise=true , address=45.91.116.12:4225 , piece_id=R2CJ5CHOJEU2R4YQHBLMSZSJME5XKIV7S4JTEJLFNON4RPUDOCFQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241488133 (59h47m6.964051173s) [noise=true , address=104.36.82.7:4909 , piece_id=BXKNHFCLRR6RFWGIIICZ3ABYJTGZR5EMPRUCLTTR7O63Y4XRUX6A, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241727270 (59h47m6.96429031s) [noise=true , address=77.90.31.10:4517 , piece_id=4NY2T5BT24RH3YY5EDXTWBXLAW5DK5FQ2SUI473B4ZCKFHDJRWTA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241780803 (59h47m6.964343843s) [noise=true , address=104.36.82.15:4259 , piece_id=OK23G7EAA4TRNKRWT2723S3QDLW7Y3H6JADC7N4PN6EEZEKSAADQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.241992398 (59h47m6.964555428s) [noise=true , address=104.36.82.2:4732 , piece_id=U7RZRKKTE2A3YT2NAXHZPGGGRXADKBJNRMY7JDR5VMDR4A27TCNQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.242236435 (59h47m6.964799475s) [noise=true , address=45.91.116.6:4975 , piece_id=2SENOK7EWBSCJXPAAHS6D4L62DGPJG67WHPGYP64ZM4PLSHCLYUA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.242446337 (59h47m6.965009367s) [noise=true , address=77.90.31.19:4842 , piece_id=DAERSSUY7VX5UXNWQR5HNT2U22WMWNNMEKGJ4UJZXVSRP7GWZ5YA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:57.242638004 (59h47m6.965201044s) [noise=true , address=104.36.82.5:4838 , piece_id=XBFMLFEPHOU56S5QATKOYJO6SOOHGW3IHM3VODDIBNB42G4XRUJA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:22:58.981229929 (59h47m8.703792959s) [upload=57357]: Next returned nil segment Jul 26 23:22:58.981251962 (59h47m8.703815002s) [upload=57357]: Waiting for error group managing segments... Jul 26 23:22:59.258281663 (59h47m8.980844703s) [noise=true , address=45.91.116.10:4083 , piece_id=MIMPJTD7WHDGTW4MS4XP3HY3SGLGKWF4OICZBU64U5M63P4DC2NQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:22:59.274332333 (59h47m8.996895373s) [noise=true , address=45.91.116.7:4011 , piece_id=NAORINYA6WT5JNHQIOXP6I5SBG4CRPPRDRBTVQVOVE6XZPFJM7FA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:22:59.337077664 (59h47m9.059640694s) [noise=true , address=45.91.116.12:4225 , piece_id=R2CJ5CHOJEU2R4YQHBLMSZSJME5XKIV7S4JTEJLFNON4RPUDOCFQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:22:59.365926510 (59h47m9.08848955s) [noise=true , address=45.91.116.6:4975 , piece_id=2SENOK7EWBSCJXPAAHS6D4L62DGPJG67WHPGYP64ZM4PLSHCLYUA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:22:59.456581532 (59h47m9.179144582s) [noise=true , address=77.90.31.15:4697 , piece_id=TNRCHE4MVDYP2K4PCXGFXJ4BT7TQLGOO3DQBRUY6DYMFBQK3K75A, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:23:00.587592219 (59h47m10.310155259s) [noise=true , address=77.90.31.18:4805 , piece_id=AP2IJBH26NFU7JQITMNVOQTQKBWEGMPB2FCIB4RJJAFNIVSRHZYQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:23:01.390453636 (59h47m11.113016676s) [noise=true , address=77.90.31.19:4842 , piece_id=DAERSSUY7VX5UXNWQR5HNT2U22WMWNNMEKGJ4UJZXVSRP7GWZ5YA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:23:26.338955588 (59h47m36.061518628s) [noise=true , address=104.36.82.10:4078 , piece_id=TRH6SMMVIMOEHYAMEWSOUUCLRD3YDWZ7AUUTWRJSXUDR3WU3M6AA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.338997900 (59h47m36.06156094s) [noise=true , address=104.36.82.3:4765 , piece_id=ISSO3GMPZOCINXDFUC35BZD3SSHLEYIKG2WFLSTETRBHFWQARHFA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339028919 (59h47m36.061591949s) [noise=true , address=104.36.82.15:4259 , piece_id=OK23G7EAA4TRNKRWT2723S3QDLW7Y3H6JADC7N4PN6EEZEKSAADQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339059547 (59h47m36.061622587s) [noise=true , address=104.36.82.2:4729 , piece_id=6GB5QRQ3R7PF5KMWJO4IOYP32WEGDI6KKDG35IYQQXP4DFKHJCKQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339091448 (59h47m36.061654478s) [noise=true , address=104.36.82.6:4873 , piece_id=3TYPHXHEXZAJFTNZZO2Z7FHA5LTYYQGGDOCGNRLK4RJULLLCJGMQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339118640 (59h47m36.06168168s) [noise=true , address=104.36.82.7:4909 , piece_id=BXKNHFCLRR6RFWGIIICZ3ABYJTGZR5EMPRUCLTTR7O63Y4XRUX6A, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339136915 (59h47m36.061699956s) [noise=true , address=77.90.31.10:4517 , piece_id=4NY2T5BT24RH3YY5EDXTWBXLAW5DK5FQ2SUI473B4ZCKFHDJRWTA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339154428 (59h47m36.061717468s) [noise=true , address=104.36.82.2:4732 , piece_id=U7RZRKKTE2A3YT2NAXHZPGGGRXADKBJNRMY7JDR5VMDR4A27TCNQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339180538 (59h47m36.061743578s) [noise=true , address=104.36.82.5:4838 , piece_id=XBFMLFEPHOU56S5QATKOYJO6SOOHGW3IHM3VODDIBNB42G4XRUJA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:37568->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:26.339209243 (59h47m36.061772293s) [seg_pos={0 0}, upload=57357]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:23:26.572151393 (59h47m36.294714433s) [upload=57357]: Done uploading segments... Jul 26 23:23:26.572210005 (59h47m36.294773085s) [upload=57357]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.234637617s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.......X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.] duration: 43.677234768s FAIL (2025-07-26 23:23:25 UTC) orbiter warning: context deadline exceeded Duration: 43.677244666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.866306812s FAIL (2025-07-26 23:23:14 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.866335265s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................................XXX.X..X.....XXXXX...] duration: 14.371021092s FAIL (2025-07-26 23:23:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.371041701s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X........X.X...........................XXXXXXXXXXXXXXXXXXX] duration: 30.2392811s FAIL (2025-07-26 23:23:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.239294816s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.245181498s FAIL (2025-07-26 23:22:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 6E7IIGSVI6QVXP7L3XMGWIYBBFQST7HKK6O73QJ2XBQ3LFVJMQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946096069 (59h45m59.668659109s) [noise=true , address=104.36.82.52:4107 , piece_id=Z5LAQNDCZY4DSJYHBY7BFCNACMLKVQWVRMBC2REOPZ7DP32FBKIQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946254011 (59h45m59.668817051s) [noise=true , address=64.49.13.140:4508 , piece_id=OFRK3PG7TDAQDUQXXKNWDFKDGEN33HTCCCSJZP6WLVIZJCUSPLTA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946371356 (59h45m59.668934396s) [noise=true , address=77.90.31.19:4875 , piece_id=Z2FCMAFAYVA2H4XVXKIFJ3A6IG7OFRW42CUC4GBV6TJKDZHLLLRA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946646292 (59h45m59.669209332s) [noise=true , address=77.90.31.17:4791 , piece_id=UC6Q6TP5M2T5JH34XXGZEM3GAUOPFLZXUV25SNEZNZB7TV4FLLCA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.946899796 (59h45m59.669462826s) [noise=true , address=104.36.82.12:4173 , piece_id=UV4ZD4DRSIY27IIV5VYR3P67KA65ZDFS5AV2WAS6WIVJ4LLA2BLA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.947131671 (59h45m59.669694711s) [noise=true , address=104.36.82.46:4511 , piece_id=4ITN25O7TMLSQSQKNS76KHAKEZGFZSZDM5XVFCXJVDCPM6N2O37A, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.947454017 (59h45m59.670017057s) [noise=true , address=79.127.205.233:4649 , piece_id=5ZOLYN57XWVFDROD3647N2AQXWLSZME3ANICO6ORHDNYXNS5OSVQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.947746757 (59h45m59.670309797s) [noise=true , address=45.91.116.13:4175 , piece_id=36ONATBRVOPS4JILW3GRKPEKDGEYD2ZFWSA2JNKDOL52CRQLWO5A, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.948009058 (59h45m59.670572088s) [noise=true , address=79.127.205.246:4260 , piece_id=6FWM6RRLYQ43MUPCQQWTTYXJLJ2EJYT7PAVEQTHMI4AIQHWOYVRQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.948335914 (59h45m59.670898954s) [noise=true , address=104.36.82.53:4150 , piece_id=2JP7PHJNM33ISTAVBRRIXRUWW7UTNMFABZAT3WU3QEM6YNFLGGPQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.948599649 (59h45m59.671162689s) [noise=true , address=79.127.205.235:4666 , piece_id=E4KJFEUHYNAAAQS7TWGPQLSGXAEG6OKXFIWXYYVNEDMKYRTPW5DQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.948845239 (59h45m59.671408279s) [noise=true , address=104.36.82.55:4210 , piece_id=2DM6B7VNRHVIMS6PQQQZ546NJQ6VCJ4W6UBBG6TVRP6XEBOVFAAQ, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.949103392 (59h45m59.671666432s) [noise=true , address=79.127.220.101:4048 , piece_id=22D5A6NLUOAL2D5ZINJFLQVE5XNLCBPJNFKFQPLCYDEHCI6UM2EA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.949420981 (59h45m59.671984021s) [noise=true , address=104.36.82.47:4526 , piece_id=BXABWY7KOKW7FMEPJCTB7PG3A2OXHVW6R7KF7DVG7WMXXKIJNLWA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.949559315 (59h45m59.672122355s) [noise=true , address=64.49.13.140:4514 , piece_id=XB2CD4WKSQ6NG5UKYCRYBWO7O5ANJB3HQCOLQCGB52TBRWFBUKIA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:49.949844120 (59h45m59.67240715s) [noise=true , address=104.36.82.54:4183 , piece_id=NS6TTWKDZ2USNM7QODIOI3NOWB2FKSPCEZQMCXLO5ESRL6NFYZPA, seg_pos={0 0}, upload=57340]: Uploading piece... Jul 26 23:21:51.056692928 (59h46m0.779255958s) [upload=57340]: Next returned nil segment Jul 26 23:21:51.056710000 (59h46m0.77927304s) [upload=57340]: Waiting for error group managing segments... Jul 26 23:21:51.353937997 (59h46m1.076501028s) [noise=true , address=79.127.205.246:4260 , piece_id=6FWM6RRLYQ43MUPCQQWTTYXJLJ2EJYT7PAVEQTHMI4AIQHWOYVRQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:51.361104319 (59h46m1.083667359s) [noise=true , address=79.127.205.235:4666 , piece_id=E4KJFEUHYNAAAQS7TWGPQLSGXAEG6OKXFIWXYYVNEDMKYRTPW5DQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:51.385354617 (59h46m1.107917657s) [noise=true , address=79.127.205.233:4649 , piece_id=5ZOLYN57XWVFDROD3647N2AQXWLSZME3ANICO6ORHDNYXNS5OSVQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:51.495253076 (59h46m1.217816116s) [noise=true , address=79.127.205.234:4597 , piece_id=QTQVIJEN7DYU7P5JR35H62GBA7JK2LZABMBJ437HVIQAZWZBR4HQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:55.807535997 (59h46m5.530099037s) [noise=true , address=79.127.220.101:4048 , piece_id=22D5A6NLUOAL2D5ZINJFLQVE5XNLCBPJNFKFQPLCYDEHCI6UM2EA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:21:59.641000414 (59h46m9.363563444s) [noise=true , address=64.49.13.140:4514 , piece_id=XB2CD4WKSQ6NG5UKYCRYBWO7O5ANJB3HQCOLQCGB52TBRWFBUKIA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:00.137462077 (59h46m9.860025107s) [noise=true , address=64.49.13.140:4508 , piece_id=OFRK3PG7TDAQDUQXXKNWDFKDGEN33HTCCCSJZP6WLVIZJCUSPLTA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:05.935782658 (59h46m15.658345698s) [noise=true , address=77.90.31.19:4875 , piece_id=Z2FCMAFAYVA2H4XVXKIFJ3A6IG7OFRW42CUC4GBV6TJKDZHLLLRA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:10.633348902 (59h46m20.355911943s) [noise=true , address=45.91.116.13:4175 , piece_id=36ONATBRVOPS4JILW3GRKPEKDGEYD2ZFWSA2JNKDOL52CRQLWO5A, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:23.241788007 (59h46m32.964351047s) [noise=true , address=79.127.201.215:4354 , piece_id=RL6E7IIGSVI6QVXP7L3XMGWIYBBFQST7HKK6O73QJ2XBQ3LFVJMQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: <nil> Jul 26 23:22:48.828518062 (59h46m58.551081102s) [noise=true , address=104.36.82.52:4107 , piece_id=Z5LAQNDCZY4DSJYHBY7BFCNACMLKVQWVRMBC2REOPZ7DP32FBKIQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: write tcp 188.166.225.170:59748->104.36.82.52:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828651207 (59h46m58.551214247s) [noise=true , address=104.36.82.53:4150 , piece_id=2JP7PHJNM33ISTAVBRRIXRUWW7UTNMFABZAT3WU3QEM6YNFLGGPQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: write tcp 188.166.225.170:43584->104.36.82.53:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828703316 (59h46m58.551266346s) [noise=true , address=104.36.82.47:4526 , piece_id=BXABWY7KOKW7FMEPJCTB7PG3A2OXHVW6R7KF7DVG7WMXXKIJNLWA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:55928->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828776877 (59h46m58.551339917s) [noise=true , address=104.36.82.54:4183 , piece_id=NS6TTWKDZ2USNM7QODIOI3NOWB2FKSPCEZQMCXLO5ESRL6NFYZPA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: write tcp 188.166.225.170:33370->104.36.82.54:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828815892 (59h46m58.551378922s) [noise=true , address=104.36.82.55:4210 , piece_id=2DM6B7VNRHVIMS6PQQQZ546NJQ6VCJ4W6UBBG6TVRP6XEBOVFAAQ, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 188.166.225.170:59992->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828855939 (59h46m58.551418969s) [noise=true , address=77.90.31.17:4791 , piece_id=UC6Q6TP5M2T5JH34XXGZEM3GAUOPFLZXUV25SNEZNZB7TV4FLLCA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: write tcp 188.166.225.170:32836->77.90.31.17:4791: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828881958 (59h46m58.551445008s) [noise=true , address=104.36.82.12:4173 , piece_id=UV4ZD4DRSIY27IIV5VYR3P67KA65ZDFS5AV2WAS6WIVJ4LLA2BLA, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:12HbGUhwH2eNqY75sLQCDgkvJ1QSrJ2n4Qg8A5Z73dFXxRpVwSC, address:104.36.82.12:4173): protocol: write tcp 188.166.225.170:56022->104.36.82.12:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828918348 (59h46m58.551481388s) [noise=true , address=104.36.82.46:4511 , piece_id=4ITN25O7TMLSQSQKNS76KHAKEZGFZSZDM5XVFCXJVDCPM6N2O37A, seg_pos={0 0}, upload=57340]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: write tcp 188.166.225.170:55300->104.36.82.46:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.828946632 (59h46m58.551509662s) [seg_pos={0 0}, upload=57340]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:22:49.072235989 (59h46m58.794799029s) [upload=57340]: Done uploading segments... Jul 26 23:22:49.072259785 (59h46m58.794822825s) [upload=57340]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.245193221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................................XXX.X..X.....XXXXX..] duration: 11.692395454s FAIL (2025-07-26 23:22:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.692421493s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [........X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX] duration: 1m0.001397937s FAIL (2025-07-26 23:22:41 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001410541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [........X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXX] duration: 30.269336184s FAIL (2025-07-26 23:22:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 9h46m11.265765536s) [upload=57344]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:01.793693011 (59h46m11.516256051s) [seg_pos={0 0}, upload=57344]: Begin upload segment... Jul 26 23:22:01.793746623 (59h46m11.516309663s) [seg_pos={0 0}, upload=57344]: Done begin upload segment. Jul 26 23:22:01.793782922 (59h46m11.516345992s) [noise=true , address=104.36.82.9:4006 , piece_id=YD765OAN7LU7DJATVJ44KU343OWU34UXE6KEMY6VMDK3IVS33NXQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.793827268 (59h46m11.516390308s) [noise=true , address=104.36.82.10:4079 , piece_id=HR4KE5RWXWM6DMTTEXAKXPEW4TXL4KZAQWYAN7USZ5N6JOJAHODQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.794225018 (59h46m11.516788058s) [noise=true , address=104.36.82.13:4185 , piece_id=S3X72RTESXC342YZKXYEUMED4R6FFTOIX5A33KNA3ENS54TVCGEQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.794300914 (59h46m11.516863954s) [noise=true , address=77.90.31.12:4589 , piece_id=OYGWROTKXGEP2EY46MSL7LK73K6JGP2P7KM7S2AD2ZXTUE7TKFHQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.794503521 (59h46m11.517066571s) [noise=true , address=104.36.82.10:4077 , piece_id=U34TRM6MAQJTCFWPMQZWI3AZ6A2ROV7BOVQPN2I6RFJMQY2YPR2Q, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.794741256 (59h46m11.517304296s) [noise=true , address=104.36.82.7:4910 , piece_id=ZLX5RCPRSVSUOBZ7WGCC3NHUDIDWXZKCVVDAAK7MUA6VWMHR33JQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795007155 (59h46m11.517570195s) [noise=true , address=104.36.82.14:4221 , piece_id=T322WMUJHZ5OIO4ZL2A33HMGB7AHPDRQ76SLARHHADAF3U6WOH3A, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795245370 (59h46m11.51780841s) [noise=true , address=104.36.82.13:4187 , piece_id=HMYYVZHWYZ4LN6DS7LMPW7FUBGKQKQZOOPVYQXENGLQHL32HA4AQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795524244 (59h46m11.518087284s) [noise=true , address=45.91.116.13:4153 , piece_id=URITHVCGSEVEDHJS2AYVJQARMA43TFGJ4HP7AFU2EYEK3WZ6J5GQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795759083 (59h46m11.518322123s) [noise=true , address=104.36.82.2:4729 , piece_id=AMDDQ3BHDJLFL5ZK7BOAUUNTBGLTHH73F5THCJENBLJQGFBA2FAQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.795840810 (59h46m11.51840386s) [noise=true , address=104.36.82.4:4803 , piece_id=RIRXNNPHUBC36SEECE2TNW2OIRJNMLQHGV6QJEC7LES73GYQ5CVA, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796097180 (59h46m11.51866022s) [noise=true , address=45.91.116.10:4083 , piece_id=WWQETFRHMNN2NGR2MUP74UR766JEATD2SZQ7EQDLWMETWCJPKCSA, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796363340 (59h46m11.51892638s) [noise=true , address=104.36.82.5:4840 , piece_id=G6IHATPVRAFEHTIKTH3ZJDD7BGGM7PD42HYYXKQN2L6TAF6TY32Q, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796475424 (59h46m11.519038464s) [noise=true , address=45.91.116.7:4009 , piece_id=RYSA7RUBXXUAXE6TPSPK66UTGDC5PM2RVGEE7RYRTTGHPERDIMPQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796714472 (59h46m11.519277512s) [noise=true , address=77.90.31.14:4661 , piece_id=H3H5UMWXQFE4Q3DV74WZCVV4W5VZ3PXBICSVFEJG3S6AGAZ76YKQ, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:01.796924333 (59h46m11.519487373s) [noise=true , address=104.36.82.7:4912 , piece_id=YF3FFPLFZDPPSEMCMM4N5UHT3LRIQ4RBDCMNFSS3PI7QKIB76EJA, seg_pos={0 0}, upload=57344]: Uploading piece... Jul 26 23:22:03.506866463 (59h46m13.229429504s) [upload=57344]: Next returned nil segment Jul 26 23:22:03.506879498 (59h46m13.229442538s) [upload=57344]: Waiting for error group managing segments... Jul 26 23:22:03.839512867 (59h46m13.562075907s) [noise=true , address=45.91.116.10:4083 , piece_id=WWQETFRHMNN2NGR2MUP74UR766JEATD2SZQ7EQDLWMETWCJPKCSA, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.859393384 (59h46m13.581956424s) [noise=true , address=104.36.82.5:4840 , piece_id=G6IHATPVRAFEHTIKTH3ZJDD7BGGM7PD42HYYXKQN2L6TAF6TY32Q, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.895081837 (59h46m13.617644897s) [noise=true , address=45.91.116.7:4009 , piece_id=RYSA7RUBXXUAXE6TPSPK66UTGDC5PM2RVGEE7RYRTTGHPERDIMPQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.899200437 (59h46m13.621763517s) [noise=true , address=77.90.31.14:4661 , piece_id=H3H5UMWXQFE4Q3DV74WZCVV4W5VZ3PXBICSVFEJG3S6AGAZ76YKQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.930337241 (59h46m13.652900281s) [noise=true , address=77.90.31.12:4589 , piece_id=OYGWROTKXGEP2EY46MSL7LK73K6JGP2P7KM7S2AD2ZXTUE7TKFHQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:03.949333096 (59h46m13.671896136s) [noise=true , address=45.91.116.13:4153 , piece_id=URITHVCGSEVEDHJS2AYVJQARMA43TFGJ4HP7AFU2EYEK3WZ6J5GQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: <nil> Jul 26 23:22:30.898358548 (59h46m40.620921588s) [noise=true , address=104.36.82.10:4079 , piece_id=HR4KE5RWXWM6DMTTEXAKXPEW4TXL4KZAQWYAN7USZ5N6JOJAHODQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898404034 (59h46m40.620967074s) [noise=true , address=104.36.82.13:4185 , piece_id=S3X72RTESXC342YZKXYEUMED4R6FFTOIX5A33KNA3ENS54TVCGEQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:36952->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898438451 (59h46m40.621001491s) [noise=true , address=104.36.82.7:4910 , piece_id=ZLX5RCPRSVSUOBZ7WGCC3NHUDIDWXZKCVVDAAK7MUA6VWMHR33JQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898474509 (59h46m40.621037549s) [noise=true , address=104.36.82.14:4221 , piece_id=T322WMUJHZ5OIO4ZL2A33HMGB7AHPDRQ76SLARHHADAF3U6WOH3A, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:33394->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898485149 (59h46m40.621048179s) [noise=true , address=104.36.82.13:4187 , piece_id=HMYYVZHWYZ4LN6DS7LMPW7FUBGKQKQZOOPVYQXENGLQHL32HA4AQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898512682 (59h46m40.621075722s) [noise=true , address=104.36.82.7:4912 , piece_id=YF3FFPLFZDPPSEMCMM4N5UHT3LRIQ4RBDCMNFSS3PI7QKIB76EJA, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898516980 (59h46m40.62108001s) [noise=true , address=104.36.82.10:4077 , piece_id=U34TRM6MAQJTCFWPMQZWI3AZ6A2ROV7BOVQPN2I6RFJMQY2YPR2Q, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898548862 (59h46m40.621111902s) [noise=true , address=104.36.82.9:4006 , piece_id=YD765OAN7LU7DJATVJ44KU343OWU34UXE6KEMY6VMDK3IVS33NXQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898559352 (59h46m40.621122392s) [noise=true , address=104.36.82.2:4729 , piece_id=AMDDQ3BHDJLFL5ZK7BOAUUNTBGLTHH73F5THCJENBLJQGFBA2FAQ, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898580753 (59h46m40.621143793s) [noise=true , address=104.36.82.4:4803 , piece_id=RIRXNNPHUBC36SEECE2TNW2OIRJNMLQHGV6QJEC7LES73GYQ5CVA, seg_pos={0 0}, upload=57344]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 188.166.225.170:49390->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:30.898605099 (59h46m40.6211682s) [seg_pos={0 0}, upload=57344]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:22:31.166289399 (59h46m40.888852439s) [upload=57344]: Done uploading segments... Jul 26 23:22:31.166316231 (59h46m40.888879271s) [upload=57344]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.269350773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X........X.X...........................XXXXXXXXXXXXXXXXXX] duration: 30.247709882s FAIL (2025-07-26 23:22:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.247741071s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.012537356s FAIL (2025-07-26 23:22:24 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.012564389s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.........X.........X...........X.......XXXXXXXXXXXXXXXXXXXXX] duration: 30.28566576s FAIL (2025-07-26 23:22:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== starting upload Jul 26 23:21:31.079201416 (59h45m40.801764486s) [upload=57334]: Uploading segments... Jul 26 23:21:31.079549612 (59h45m40.802112672s) [upload=57334]: Got next segment. Inline: false Jul 26 23:21:31.079599757 (59h45m40.802162827s) [upload=57334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:21:31.079607994 (59h45m40.802171064s) [upload=57334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:21:31.315397450 (59h45m41.03796049s) [seg_pos={0 0}, upload=57334]: Begin upload segment... Jul 26 23:21:31.315446834 (59h45m41.038009874s) [seg_pos={0 0}, upload=57334]: Done begin upload segment. Jul 26 23:21:31.315466151 (59h45m41.038029191s) [noise=true , address=77.90.31.19:4842 , piece_id=6U6HGX3ZIMQNWRAMOJGCDXSI6UR2C3XLOBQ47IRMO22NFIW6FGFQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.315475268 (59h45m41.038038308s) [noise=true , address=45.91.116.5:4297 , piece_id=D344A2TZZZ6GCOKYHCSLEYWK4GXOSZINHLZ3SGOXYIF65JRN4BSA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.315796934 (59h45m41.038359974s) [noise=true , address=104.36.82.12:4149 , piece_id=IG4VMFEDBIOVAXJKNGM76AEOSRIDASTZ63AFR5SD3ZPVMMYBLQPQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.316041151 (59h45m41.038604192s) [noise=true , address=104.36.82.2:4729 , piece_id=6FUK3H5KAVWFFFZQQCQJVR3IBXYLEVWPKMCPMG7LNKQK6HUT333Q, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.316315496 (59h45m41.038878536s) [noise=true , address=104.36.82.7:4910 , piece_id=WTOXS2R5UINBYNC6RL3CPCIJ5JVGOZHISLKKPSM7ISHKT44S36BA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.316553933 (59h45m41.039116963s) [noise=true , address=77.90.31.14:4661 , piece_id=Q4MKNDWIPQBIRFUFBDRWZRFLEPHZRML5JOWVQJLEYIG2S2DDYCWA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.316786688 (59h45m41.039349728s) [noise=true , address=104.36.82.9:4007 , piece_id=JZM6F4MPJSO4TOGKUENSMJWPIKWO72Y2RLJM6R6DCN5XKOEGCHCA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.317557232 (59h45m41.040120272s) [noise=true , address=45.91.116.9:4261 , piece_id=MGKCTJ6ZIIGXEJOYAEEYFZCMV2FBZVMQ33HUSAV3KZXHKZXSJ55A, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.317973308 (59h45m41.040536348s) [noise=true , address=104.36.82.5:4837 , piece_id=OZ5KWYRELI74XCVPZ6VHSXH2GHH5X6QSZWS6OBUQDS6UZRLMQXSQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.318438448 (59h45m41.041001488s) [noise=true , address=45.91.116.14:4191 , piece_id=HGLJN2HB7IEYSBNSYZ5T2TKR4SI5ZJ5OELRJXW3HA4R3XHSWGSZA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.318793447 (59h45m41.041356487s) [noise=true , address=104.36.82.10:4077 , piece_id=H6EHHMERH5Q6D7FRUOOE26YWFA5I4YA3KZDG2XW4JVWWSKH2UWEQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319037494 (59h45m41.041600534s) [noise=true , address=104.36.82.14:4222 , piece_id=VEUQZ32IOANF4ETMPAPOF5ZEERJMBF6DSYIBIIE6OHBAPIHGVBMA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319327840 (59h45m41.04189088s) [noise=true , address=104.36.82.9:4005 , piece_id=EERIWAPGKC7TE7BO3OI3F7UPIVJZWYGRTCGHB3DBSDWS6HOJQYVQ, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319558030 (59h45m41.04212106s) [noise=true , address=45.91.116.13:4154 , piece_id=O2PJG7XBIRJHVT6JMI3KGVF5666GYNIB3KD7YDD25D5H6ZMLON7Q, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319779394 (59h45m41.042342434s) [noise=true , address=104.36.82.13:4186 , piece_id=5GY2XWA7XVCW67BLQFLYFEMJKWNNARVAHXGPNAI6IP67V6QZQMAA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:31.319993223 (59h45m41.042556263s) [noise=true , address=104.36.82.4:4802 , piece_id=7ZDQEMVP7H3SAI425NWBF2EM73HM43UHSRUWECTSSHA5ICV7OQCA, seg_pos={0 0}, upload=57334]: Uploading piece... Jul 26 23:21:33.025563649 (59h45m42.748126689s) [upload=57334]: Next returned nil segment Jul 26 23:21:33.025582465 (59h45m42.748145505s) [upload=57334]: Waiting for error group managing segments... Jul 26 23:21:33.340809481 (59h45m43.063372521s) [noise=true , address=77.90.31.14:4661 , piece_id=Q4MKNDWIPQBIRFUFBDRWZRFLEPHZRML5JOWVQJLEYIG2S2DDYCWA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.409544172 (59h45m43.132107202s) [noise=true , address=45.91.116.13:4154 , piece_id=O2PJG7XBIRJHVT6JMI3KGVF5666GYNIB3KD7YDD25D5H6ZMLON7Q, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.416314225 (59h45m43.138877255s) [noise=true , address=104.36.82.5:4837 , piece_id=OZ5KWYRELI74XCVPZ6VHSXH2GHH5X6QSZWS6OBUQDS6UZRLMQXSQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.436711050 (59h45m43.15927409s) [noise=true , address=45.91.116.5:4297 , piece_id=D344A2TZZZ6GCOKYHCSLEYWK4GXOSZINHLZ3SGOXYIF65JRN4BSA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.497740649 (59h45m43.220303689s) [noise=true , address=45.91.116.14:4191 , piece_id=HGLJN2HB7IEYSBNSYZ5T2TKR4SI5ZJ5OELRJXW3HA4R3XHSWGSZA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:33.843048748 (59h45m43.565611788s) [noise=true , address=45.91.116.9:4261 , piece_id=MGKCTJ6ZIIGXEJOYAEEYFZCMV2FBZVMQ33HUSAV3KZXHKZXSJ55A, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:21:35.443196869 (59h45m45.165759909s) [noise=true , address=77.90.31.19:4842 , piece_id=6U6HGX3ZIMQNWRAMOJGCDXSI6UR2C3XLOBQ47IRMO22NFIW6FGFQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: <nil> Jul 26 23:22:00.377926558 (59h46m10.100489598s) [noise=true , address=104.36.82.12:4149 , piece_id=IG4VMFEDBIOVAXJKNGM76AEOSRIDASTZ63AFR5SD3ZPVMMYBLQPQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.377988256 (59h46m10.100551296s) [noise=true , address=104.36.82.2:4729 , piece_id=6FUK3H5KAVWFFFZQQCQJVR3IBXYLEVWPKMCPMG7LNKQK6HUT333Q, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378051046 (59h46m10.100614086s) [noise=true , address=104.36.82.7:4910 , piece_id=WTOXS2R5UINBYNC6RL3CPCIJ5JVGOZHISLKKPSM7ISHKT44S36BA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378113666 (59h46m10.100676706s) [noise=true , address=104.36.82.9:4007 , piece_id=JZM6F4MPJSO4TOGKUENSMJWPIKWO72Y2RLJM6R6DCN5XKOEGCHCA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378190132 (59h46m10.100753172s) [noise=true , address=104.36.82.10:4077 , piece_id=H6EHHMERH5Q6D7FRUOOE26YWFA5I4YA3KZDG2XW4JVWWSKH2UWEQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378225520 (59h46m10.10078856s) [noise=true , address=104.36.82.4:4802 , piece_id=7ZDQEMVP7H3SAI425NWBF2EM73HM43UHSRUWECTSSHA5ICV7OQCA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378296996 (59h46m10.100860037s) [noise=true , address=104.36.82.14:4222 , piece_id=VEUQZ32IOANF4ETMPAPOF5ZEERJMBF6DSYIBIIE6OHBAPIHGVBMA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378355839 (59h46m10.100918879s) [noise=true , address=104.36.82.13:4186 , piece_id=5GY2XWA7XVCW67BLQFLYFEMJKWNNARVAHXGPNAI6IP67V6QZQMAA, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:00.378413600 (59h46m10.10097664s) [noise=true , address=104.36.82.9:4005 , piece_id=EERIWAPGKC7TE7BO3OI3F7UPIVJZWYGRTCGHB3DBSDWS6HOJQYVQ, seg_pos={0 0}, upload=57334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:57714->104.36.82.9:4005: use of closed network connection; context deadline exceeded Jul 26 23:22:00.378447435 (59h46m10.101010465s) [seg_pos={0 0}, upload=57334]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:22:00.661793613 (59h46m10.384356653s) [upload=57334]: Done uploading segments... Jul 26 23:22:00.661831605 (59h46m10.384394645s) [upload=57334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.285686891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.250758726s FAIL (2025-07-26 23:21:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== YZCV5E3D4FTDJDQ4AOBOLPZG46A, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.426412636 (59h44m59.148975666s) [noise=true , address=104.36.82.56:4238 , piece_id=2L3EKIWVFPAWWPDXS6LSYMLFFWKSOHPG7KCJZWDSGOMPNG7FIN7Q, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.426760120 (59h44m59.14932315s) [noise=true , address=77.90.31.16:4748 , piece_id=A6QIUUU6AXSMLL3A4SG42XXM37TSP3TIZ74NFBHVGJTE6ARIMUBQ, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.426828191 (59h44m59.149391231s) [noise=true , address=104.36.82.52:4089 , piece_id=32FYUVUECR7TBXWPTLKCXB5RCP3G3D4HMSSLYWFTVEC2WJOZK3OA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427024156 (59h44m59.149587196s) [noise=true , address=79.127.201.214:4321 , piece_id=OJYW4S5XCZRR6QFLYQPJ77YJQSGFLEINUKB4YSXT4G6QFNQACTPA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427129037 (59h44m59.149692077s) [noise=true , address=77.90.31.11:4572 , piece_id=5BYPH5YFW3TWDNAI2IADIBDT7JV6CWZKNWXYDHZAJFWVQT7HMH2Q, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427297279 (59h44m59.149860309s) [noise=true , address=64.49.13.140:4514 , piece_id=XE3BJHE3W3YIM2E2GRRIKIRIJOTQZLAGQSMRQ2KAICZF37HD3JNA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427527769 (59h44m59.150090809s) [noise=true , address=45.91.116.9:4267 , piece_id=XV3IOMBNZNOIYDA5B34ZU2N6RY6OCJH2LCOBSBWMS6LNWDQ52URA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427752118 (59h44m59.150315158s) [noise=true , address=79.127.219.51:4212 , piece_id=PNIJ732N6HDEIMJFTTXFUHW4RMXZLLXHYAP2L37DE76I7WQB2SLA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.427973913 (59h44m59.150536953s) [noise=true , address=79.127.205.235:4688 , piece_id=WHVNI3X3I7N7MUFMMDNXRLZUGIFXFL6XTMTGOFM2SJ2ALDBOLIGA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.428233259 (59h44m59.150796309s) [noise=true , address=79.127.219.38:4003 , piece_id=FQZGA42QM7V6Z6FD3ODTN6U6MOHDY35A67SL74EDANY5KL42JHBA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.428439674 (59h44m59.151002714s) [noise=true , address=104.36.82.46:4504 , piece_id=RUOISBTRTRKLA7FNPJC4KSCJJLMBGCP7U5VLQGEEZPYPGZV5D4JQ, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.428635308 (59h44m59.151198348s) [noise=true , address=79.127.205.245:4242 , piece_id=R7JDMGKT4Q3EJMC5IVWZHMWXZ2ETKESQIUIKCWBHFF6GRECNIDWQ, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.428820903 (59h44m59.151383943s) [noise=true , address=79.127.205.246:4258 , piece_id=L5RVEHM62RHOJUXJUUJTROZC5N5IRGRV7WEEIX36DX23Q6OGUPWA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.429448174 (59h44m59.152011214s) [noise=true , address=77.90.31.10:4531 , piece_id=CJLMHTTDAGPC2T6JHVSFIDE2QDNUEVEU73TUR6B52HJS3G3HGPKA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.429840203 (59h44m59.152403243s) [noise=true , address=104.36.82.54:4169 , piece_id=RDIWB7MIGGU3YAGGEGYJWQPA7YJQEB4DPT7KKVWHH6AGKZOTGOWA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.430174423 (59h44m59.152737463s) [noise=true , address=79.127.205.234:4614 , piece_id=XMQ3ZN6HQLNZW3HGKKPKQOBVXU2GCM2OSUR3F5J75AE6IRPRAWZA, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:49.430533580 (59h44m59.15309662s) [noise=true , address=77.90.31.13:4629 , piece_id=W3XIJEG3YRZMJDW3L4LJ6DQ7ZE3VKJTJ6TDAAQC4FZE7I7SBWAKQ, seg_pos={0 0}, upload=57324]: Uploading piece... Jul 26 23:20:50.514758382 (59h45m0.237321422s) [upload=57324]: Next returned nil segment Jul 26 23:20:50.514777229 (59h45m0.237340269s) [upload=57324]: Waiting for error group managing segments... Jul 26 23:20:50.802192692 (59h45m0.524755732s) [noise=true , address=79.127.205.234:4614 , piece_id=XMQ3ZN6HQLNZW3HGKKPKQOBVXU2GCM2OSUR3F5J75AE6IRPRAWZA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:50.840550742 (59h45m0.563113782s) [noise=true , address=79.127.205.246:4258 , piece_id=L5RVEHM62RHOJUXJUUJTROZC5N5IRGRV7WEEIX36DX23Q6OGUPWA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:50.840862509 (59h45m0.563425549s) [noise=true , address=79.127.205.235:4688 , piece_id=WHVNI3X3I7N7MUFMMDNXRLZUGIFXFL6XTMTGOFM2SJ2ALDBOLIGA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:51.222771477 (59h45m0.945334517s) [noise=true , address=79.127.205.245:4242 , piece_id=R7JDMGKT4Q3EJMC5IVWZHMWXZ2ETKESQIUIKCWBHFF6GRECNIDWQ, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:51.407900064 (59h45m1.130463104s) [noise=true , address=79.127.219.38:4003 , piece_id=FQZGA42QM7V6Z6FD3ODTN6U6MOHDY35A67SL74EDANY5KL42JHBA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:51.434483416 (59h45m1.157046456s) [noise=true , address=77.90.31.11:4572 , piece_id=5BYPH5YFW3TWDNAI2IADIBDT7JV6CWZKNWXYDHZAJFWVQT7HMH2Q, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:54.067570544 (59h45m3.790133584s) [noise=true , address=79.127.219.51:4212 , piece_id=PNIJ732N6HDEIMJFTTXFUHW4RMXZLLXHYAP2L37DE76I7WQB2SLA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:54.849175391 (59h45m4.571738431s) [noise=true , address=79.127.201.214:4321 , piece_id=OJYW4S5XCZRR6QFLYQPJ77YJQSGFLEINUKB4YSXT4G6QFNQACTPA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:20:56.418457887 (59h45m6.141020927s) [noise=true , address=45.91.116.9:4267 , piece_id=XV3IOMBNZNOIYDA5B34ZU2N6RY6OCJH2LCOBSBWMS6LNWDQ52URA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:21:00.082920028 (59h45m9.805483079s) [noise=true , address=64.49.13.140:4514 , piece_id=XE3BJHE3W3YIM2E2GRRIKIRIJOTQZLAGQSMRQ2KAICZF37HD3JNA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: <nil> Jul 26 23:21:48.337414289 (59h45m58.059977329s) [noise=true , address=77.90.31.13:4629 , piece_id=W3XIJEG3YRZMJDW3L4LJ6DQ7ZE3VKJTJ6TDAAQC4FZE7I7SBWAKQ, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:21:48.337692752 (59h45m58.060255792s) [noise=true , address=77.90.31.10:4531 , piece_id=CJLMHTTDAGPC2T6JHVSFIDE2QDNUEVEU73TUR6B52HJS3G3HGPKA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: rpc: dial tcp 77.90.31.10:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337723180 (59h45m58.060286221s) [noise=true , address=104.36.82.46:4504 , piece_id=RUOISBTRTRKLA7FNPJC4KSCJJLMBGCP7U5VLQGEEZPYPGZV5D4JQ, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:47886->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337747237 (59h45m58.060310277s) [noise=true , address=104.36.82.47:4529 , piece_id=SNZOK5DIYDWTXV3NU2KZRZQ6MYZCV5E3D4FTDJDQ4AOBOLPZG46A, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:1m7FrQEps7d8enRYk4v17UxcDs8o8L2CFcVEyB3zR4EniXrYeH, address:104.36.82.47:4529): protocol: write tcp 188.166.225.170:50350->104.36.82.47:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337781702 (59h45m58.060344742s) [noise=true , address=104.36.82.52:4089 , piece_id=32FYUVUECR7TBXWPTLKCXB5RCP3G3D4HMSSLYWFTVEC2WJOZK3OA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 188.166.225.170:53460->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337835235 (59h45m58.060398275s) [noise=true , address=104.36.82.56:4238 , piece_id=2L3EKIWVFPAWWPDXS6LSYMLFFWKSOHPG7KCJZWDSGOMPNG7FIN7Q, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: write tcp 188.166.225.170:42094->104.36.82.56:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337844202 (59h45m58.060407242s) [noise=true , address=104.36.82.54:4169 , piece_id=RDIWB7MIGGU3YAGGEGYJWQPA7YJQEB4DPT7KKVWHH6AGKZOTGOWA, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337894598 (59h45m58.060457628s) [noise=true , address=77.90.31.16:4748 , piece_id=A6QIUUU6AXSMLL3A4SG42XXM37TSP3TIZ74NFBHVGJTE6ARIMUBQ, seg_pos={0 0}, upload=57324]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: write tcp 188.166.225.170:48038->77.90.31.16:4748: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:48.337915729 (59h45m58.060478759s) [seg_pos={0 0}, upload=57324]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:21:48.586441184 (59h45m58.309004214s) [upload=57324]: Done uploading segments... Jul 26 23:21:48.586468096 (59h45m58.309031136s) [upload=57324]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.250782823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.........X..XX...X.X.X.X.......XX..XX.XX..XXX............XXX] duration: 1m0.001107154s FAIL (2025-07-26 23:21:41 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001118706s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................................XXX.X..X.....XXXXX.] duration: 5.980909552s FAIL (2025-07-26 23:21:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.980922528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X........X.X...........................XXXXXXXXXXXXXXXXX] duration: 30.235871814s FAIL (2025-07-26 23:21:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.235893866s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.812039486s FAIL (2025-07-26 23:21:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.812064172s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................................XXX.X..X.....XXXXX] duration: 30.001325926s FAIL (2025-07-26 23:21:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00133809s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.855342493s FAIL (2025-07-26 23:21:03 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.855369345s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..........X.........X...........X.......XXXXXXXXXXXXXXXXXXXX] duration: 30.232254825s FAIL (2025-07-26 23:20:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== .609784210 (59h44m36.33234725s) [seg_pos={0 0}, upload=57319]: Done begin upload segment. Jul 26 23:20:26.609840948 (59h44m36.332403988s) [noise=true , address=104.36.82.13:4186 , piece_id=2336GFAUBVRMN264ZXVBDTAEDQH35KZFO3RWC6F6ZQRALPSUYDIQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.609893960 (59h44m36.332457s) [noise=true , address=104.36.82.6:4873 , piece_id=F6DOAHCUKRLZPLRFX7RXDVOX7KZ3X4CO2AD75CUG7M556RPXXARA, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.609970707 (59h44m36.332533747s) [noise=true , address=45.91.116.10:4083 , piece_id=BTKW3WUU4JRQVOQPORL3XLGWXUYQEBQ2UGJKDD2PZFCJSD6LDS3Q, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.610278956 (59h44m36.332841996s) [noise=true , address=104.36.82.9:4006 , piece_id=STVI6L2Z7GCJ5WCNGBVQMONMFWWFQVKWWZOYSL2KSKNLBMWWV46A, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.610388185 (59h44m36.332951225s) [noise=true , address=104.36.82.15:4259 , piece_id=YE2CPT7F7UTJNKKZUHKVH2NCRI6SU3FPOYYUZAX5L7SKRX324LKQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.610585444 (59h44m36.333148484s) [noise=true , address=45.91.116.14:4190 , piece_id=XM5LWA4E625UC76WUE3AZXVPSPXWTT4RZUCKW4YIBJVMTH5UXSHA, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.610662100 (59h44m36.33322514s) [noise=true , address=45.91.116.12:4226 , piece_id=3QRFH54R5BDYK4N2WCTZQKTUMAJZU2MJZGJWVMZ6EZHKDL65QWJQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.610830492 (59h44m36.333393532s) [noise=true , address=104.36.82.16:4294 , piece_id=KDZV6TGXIFFYBNVTQ4KCIUAMNETTADDGFJMIRBDRVDGOV735HN6A, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.610935252 (59h44m36.333498292s) [noise=true , address=104.36.82.3:4765 , piece_id=ZHG32QZ4XU7SVYGHVRSH3VLMJHGEAE34GBXLRDYPAUDUMIIKQKMQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.611073076 (59h44m36.333636116s) [noise=true , address=45.91.116.4:4940 , piece_id=OBC6JWYCN2F3CPCEIOAPYW2DDKAYA75GCUSUADOMHJDKKAHNNUFQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.611212302 (59h44m36.333775342s) [noise=true , address=45.91.116.13:4155 , piece_id=VK3H7UK6ZFPLOULLZ6F7M5IJVV5FJRAPOCMRLXXZTOGQXSYSUAWQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.611312314 (59h44m36.333875354s) [noise=true , address=104.36.82.2:4730 , piece_id=PFKORENNDEX4CSQUJG2R3Z2WWNBJ2DHU2FNDRZB2CERQZ25S4SBQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.611476578 (59h44m36.334039618s) [noise=true , address=104.36.82.5:4840 , piece_id=QYFJLCZWTZNHNFQO3L3MWQBBLNOEK34JGOMLV3BHHHBQHKRCPXXQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.611555680 (59h44m36.33411872s) [noise=true , address=104.36.82.10:4079 , piece_id=WE3TR27IYU2D4QRP7YBYZJIIVHLRFJM7INLHFQE7EW2T4BK4Q3DA, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.611690267 (59h44m36.334253307s) [noise=true , address=104.36.82.11:4115 , piece_id=URXUCJJ22JCCOJESOMCG6ZTOPZ5DL7CIOEEFZ5Y4R5UKQT52PQEQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:26.611761974 (59h44m36.334325024s) [noise=true , address=104.36.82.7:4909 , piece_id=BPV7OU6LJXLOPKEQ4VGZVKT5DLVQCIQ5R4SFIC7PMKXLBWK4AKWQ, seg_pos={0 0}, upload=57319]: Uploading piece... Jul 26 23:20:28.364548448 (59h44m38.087111498s) [upload=57319]: Next returned nil segment Jul 26 23:20:28.364570420 (59h44m38.08713346s) [upload=57319]: Waiting for error group managing segments... Jul 26 23:20:28.642361298 (59h44m38.364924338s) [noise=true , address=45.91.116.14:4190 , piece_id=XM5LWA4E625UC76WUE3AZXVPSPXWTT4RZUCKW4YIBJVMTH5UXSHA, seg_pos={0 0}, upload=57319]: Done uploading piece. err: <nil> Jul 26 23:20:28.647556709 (59h44m38.370119749s) [noise=true , address=45.91.116.4:4940 , piece_id=OBC6JWYCN2F3CPCEIOAPYW2DDKAYA75GCUSUADOMHJDKKAHNNUFQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: <nil> Jul 26 23:20:28.690151786 (59h44m38.412714826s) [noise=true , address=45.91.116.10:4083 , piece_id=BTKW3WUU4JRQVOQPORL3XLGWXUYQEBQ2UGJKDD2PZFCJSD6LDS3Q, seg_pos={0 0}, upload=57319]: Done uploading piece. err: <nil> Jul 26 23:20:28.721531315 (59h44m38.444094355s) [noise=true , address=45.91.116.13:4155 , piece_id=VK3H7UK6ZFPLOULLZ6F7M5IJVV5FJRAPOCMRLXXZTOGQXSYSUAWQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: <nil> Jul 26 23:20:28.728966250 (59h44m38.45152929s) [noise=true , address=45.91.116.12:4226 , piece_id=3QRFH54R5BDYK4N2WCTZQKTUMAJZU2MJZGJWVMZ6EZHKDL65QWJQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: <nil> Jul 26 23:20:55.730613881 (59h45m5.453176921s) [noise=true , address=104.36.82.13:4186 , piece_id=2336GFAUBVRMN264ZXVBDTAEDQH35KZFO3RWC6F6ZQRALPSUYDIQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.730661953 (59h45m5.453224993s) [noise=true , address=104.36.82.9:4006 , piece_id=STVI6L2Z7GCJ5WCNGBVQMONMFWWFQVKWWZOYSL2KSKNLBMWWV46A, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.730717519 (59h45m5.453280559s) [noise=true , address=104.36.82.15:4259 , piece_id=YE2CPT7F7UTJNKKZUHKVH2NCRI6SU3FPOYYUZAX5L7SKRX324LKQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:41598->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.730746755 (59h45m5.453309785s) [noise=true , address=104.36.82.10:4079 , piece_id=WE3TR27IYU2D4QRP7YBYZJIIVHLRFJM7INLHFQE7EW2T4BK4Q3DA, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.730791732 (59h45m5.453354772s) [noise=true , address=104.36.82.7:4909 , piece_id=BPV7OU6LJXLOPKEQ4VGZVKT5DLVQCIQ5R4SFIC7PMKXLBWK4AKWQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:38534->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.730940275 (59h45m5.453503315s) [noise=true , address=104.36.82.6:4873 , piece_id=F6DOAHCUKRLZPLRFX7RXDVOX7KZ3X4CO2AD75CUG7M556RPXXARA, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.731036740 (59h45m5.453599781s) [noise=true , address=104.36.82.16:4294 , piece_id=KDZV6TGXIFFYBNVTQ4KCIUAMNETTADDGFJMIRBDRVDGOV735HN6A, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.731078800 (59h45m5.45364184s) [noise=true , address=104.36.82.2:4730 , piece_id=PFKORENNDEX4CSQUJG2R3Z2WWNBJ2DHU2FNDRZB2CERQZ25S4SBQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.731099049 (59h45m5.453662089s) [noise=true , address=104.36.82.11:4115 , piece_id=URXUCJJ22JCCOJESOMCG6ZTOPZ5DL7CIOEEFZ5Y4R5UKQT52PQEQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.731122775 (59h45m5.453685815s) [noise=true , address=104.36.82.5:4840 , piece_id=QYFJLCZWTZNHNFQO3L3MWQBBLNOEK34JGOMLV3BHHHBQHKRCPXXQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.731139446 (59h45m5.453702476s) [noise=true , address=104.36.82.3:4765 , piece_id=ZHG32QZ4XU7SVYGHVRSH3VLMJHGEAE34GBXLRDYPAUDUMIIKQKMQ, seg_pos={0 0}, upload=57319]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:58394->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:55.731154395 (59h45m5.453717425s) [seg_pos={0 0}, upload=57319]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:20:55.961179494 (59h45m5.683742534s) [upload=57319]: Done uploading segments... Jul 26 23:20:55.961201105 (59h45m5.683764145s) [upload=57319]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.232291596s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..........X..XX...X.X.X.X.......XX..XX.XX..XXX............XX] duration: 1m0.00066416s FAIL (2025-07-26 23:20:41 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000677587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232357993s FAIL (2025-07-26 23:20:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== e begin upload segment. Jul 26 23:19:40.444295931 (59h43m50.166858961s) [noise=true , address=104.36.82.54:4161 , piece_id=CBHYZXNYH33BTLQO5BJ4UETO7NU2XBC7PLCXLKJZSP64S5T5RU6A, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.444345957 (59h43m50.166909027s) [noise=true , address=79.127.205.234:4610 , piece_id=7B44AAPDNKOP2VSJSFXL5OX2VMY5ANJKFS2GNEFD6FQJGRRX3OTQ, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.444636824 (59h43m50.167199854s) [noise=true , address=104.36.82.53:4132 , piece_id=VD7MODVTMBJEFDCTYY5EGDO6HFAORMDHSUYZGYKKJF3SSFTKBC5Q, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.444872404 (59h43m50.167435444s) [noise=true , address=77.90.31.21:4886 , piece_id=IXC6VVQM2OAYLMQNLHJ6TF66GRRGXK6RYCSKTCCHNQ57IIWUHI2A, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.444891581 (59h43m50.167454621s) [noise=true , address=104.36.82.46:4514 , piece_id=HNOTMF5IHZ626LZ5QMKXDX573JFGQUQY6LVJNEKYEPDWLYVVWQ7A, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.445132563 (59h43m50.167695593s) [noise=true , address=77.90.31.13:4647 , piece_id=D5BSLIVIPQFICXK5JJTOUCZU7IBT3QJAQ4JXL2G3OQ5E4AIXMPCA, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.445159975 (59h43m50.167723015s) [noise=true , address=77.90.31.15:4730 , piece_id=66S5KQJXKI5TTXYCSQ27ONCGRX3BUMFVLLYJA7ASZUO632TBW6PA, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.445370327 (59h43m50.167933367s) [noise=true , address=64.49.13.138:4482 , piece_id=N7CCG5WW5G4INJCV2N2ZAO7EZ52MJMD6B3HAIDIKUWGTDCUSYAWA, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.445613923 (59h43m50.168176963s) [noise=true , address=79.127.205.245:4251 , piece_id=V6MGDYONI23ITEWQPTZMHLCKKG7WPEBM6KHJSRTEAXOCKNYP56NQ, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.445831359 (59h43m50.168394399s) [noise=true , address=104.36.82.56:4234 , piece_id=3JMBBWWRRPAO7I224FTJTQ4TYKLFOJTMFPDF446NVZ55IHDUZDHA, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.446029819 (59h43m50.168592859s) [noise=true , address=79.127.201.215:4360 , piece_id=X3VTHTCLCDYDKQ5IOU6RHAHT73XNJS5LS2JSF4V5TX5ZVHSAHS5Q, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.446225172 (59h43m50.168788212s) [noise=true , address=79.127.205.233:4645 , piece_id=GF6OE6AHC4DGOT2JT5LRRTIA6FIXLXLT57S7JCFOVK5ZJLNBFXBQ, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.446426749 (59h43m50.168989779s) [noise=true , address=104.36.82.47:4541 , piece_id=QTIIOOLGOQAJ4N6QJTMCG7KFD4X6CPTAVS6HVDYEWZGZED7ARCLQ, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.446646429 (59h43m50.169209469s) [noise=true , address=79.127.219.38:4007 , piece_id=U6T7JFKLOOF5WZYDLUOVWICNFW6FNW6JRGIBFGB3WMR33GNCGHHQ, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.446979727 (59h43m50.169542757s) [noise=true , address=104.36.82.52:4112 , piece_id=4MMFVAPABI2L3TVHLTN5MI4OSVWKHB2PHN2KBZLMRZPTG2OKOS4A, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.447025014 (59h43m50.169588054s) [noise=true , address=77.90.31.19:4860 , piece_id=FL2DDHBXDN72K46BEGYCSIZPTQT2CP4QWC5OFEHOQVRP4OQOJF6A, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.447256656 (59h43m50.169819696s) [noise=true , address=79.127.219.51:4212 , piece_id=TQTRAT2TWXBPL2U5CHWNE5JWQWNUN2FXKTQSEBOU6PNXKRUVTVSA, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:40.448069793 (59h43m50.170632833s) [noise=true , address=104.36.82.38:4690 , piece_id=MGTMK33Z3MMHM4Z6O6DA5HL7TV7QKRAP2T4T6PVDR6JLMOQ7ZRCA, seg_pos={0 0}, upload=57305]: Uploading piece... Jul 26 23:19:41.461653007 (59h43m51.184216047s) [noise=true , address=64.49.13.138:4482 , piece_id=N7CCG5WW5G4INJCV2N2ZAO7EZ52MJMD6B3HAIDIKUWGTDCUSYAWA, seg_pos={0 0}, upload=57305]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:19:41.565652530 (59h43m51.28821556s) [upload=57305]: Next returned nil segment Jul 26 23:19:41.565684871 (59h43m51.288247911s) [upload=57305]: Waiting for error group managing segments... Jul 26 23:19:41.855767718 (59h43m51.578330758s) [noise=true , address=79.127.205.233:4645 , piece_id=GF6OE6AHC4DGOT2JT5LRRTIA6FIXLXLT57S7JCFOVK5ZJLNBFXBQ, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:19:42.093481176 (59h43m51.816044206s) [noise=true , address=79.127.205.245:4251 , piece_id=V6MGDYONI23ITEWQPTZMHLCKKG7WPEBM6KHJSRTEAXOCKNYP56NQ, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:19:42.149171237 (59h43m51.871734267s) [noise=true , address=79.127.205.234:4610 , piece_id=7B44AAPDNKOP2VSJSFXL5OX2VMY5ANJKFS2GNEFD6FQJGRRX3OTQ, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:19:42.327301154 (59h43m52.049864194s) [noise=true , address=79.127.219.38:4007 , piece_id=U6T7JFKLOOF5WZYDLUOVWICNFW6FNW6JRGIBFGB3WMR33GNCGHHQ, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:19:42.434149728 (59h43m52.156712768s) [noise=true , address=77.90.31.21:4886 , piece_id=IXC6VVQM2OAYLMQNLHJ6TF66GRRGXK6RYCSKTCCHNQ57IIWUHI2A, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:19:44.162540566 (59h43m53.885103596s) [noise=true , address=79.127.219.51:4212 , piece_id=TQTRAT2TWXBPL2U5CHWNE5JWQWNUN2FXKTQSEBOU6PNXKRUVTVSA, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:19:48.722234674 (59h43m58.444797714s) [noise=true , address=77.90.31.13:4647 , piece_id=D5BSLIVIPQFICXK5JJTOUCZU7IBT3QJAQ4JXL2G3OQ5E4AIXMPCA, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:19:56.774345095 (59h44m6.496908135s) [noise=true , address=77.90.31.19:4860 , piece_id=FL2DDHBXDN72K46BEGYCSIZPTQT2CP4QWC5OFEHOQVRP4OQOJF6A, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:20:00.258833473 (59h44m9.981396503s) [noise=true , address=79.127.201.215:4360 , piece_id=X3VTHTCLCDYDKQ5IOU6RHAHT73XNJS5LS2JSF4V5TX5ZVHSAHS5Q, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:20:38.073614728 (59h44m47.796177758s) [noise=true , address=77.90.31.15:4730 , piece_id=66S5KQJXKI5TTXYCSQ27ONCGRX3BUMFVLLYJA7ASZUO632TBW6PA, seg_pos={0 0}, upload=57305]: Done uploading piece. err: <nil> Jul 26 23:20:39.342930756 (59h44m49.065493796s) [noise=true , address=104.36.82.38:4690 , piece_id=MGTMK33Z3MMHM4Z6O6DA5HL7TV7QKRAP2T4T6PVDR6JLMOQ7ZRCA, seg_pos={0 0}, upload=57305]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:104.36.82.38:4690): protocol: rpc: dial tcp 104.36.82.38:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:39.342971454 (59h44m49.065534494s) [noise=true , address=104.36.82.47:4541 , piece_id=QTIIOOLGOQAJ4N6QJTMCG7KFD4X6CPTAVS6HVDYEWZGZED7ARCLQ, seg_pos={0 0}, upload=57305]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:39.343001671 (59h44m49.065564711s) [noise=true , address=104.36.82.54:4161 , piece_id=CBHYZXNYH33BTLQO5BJ4UETO7NU2XBC7PLCXLKJZSP64S5T5RU6A, seg_pos={0 0}, upload=57305]: Done uploading piece. err: ecclient: upload failed (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD, address:104.36.82.54:4161): protocol: rpc: dial tcp 104.36.82.54:4161: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:39.343035877 (59h44m49.065598907s) [noise=true , address=104.36.82.53:4132 , piece_id=VD7MODVTMBJEFDCTYY5EGDO6HFAORMDHSUYZGYKKJF3SSFTKBC5Q, seg_pos={0 0}, upload=57305]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:39.343063680 (59h44m49.06562672s) [noise=true , address=104.36.82.46:4514 , piece_id=HNOTMF5IHZ626LZ5QMKXDX573JFGQUQY6LVJNEKYEPDWLYVVWQ7A, seg_pos={0 0}, upload=57305]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: rpc: dial tcp 104.36.82.46:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:39.343097485 (59h44m49.065660525s) [noise=true , address=104.36.82.56:4234 , piece_id=3JMBBWWRRPAO7I224FTJTQ4TYKLFOJTMFPDF446NVZ55IHDUZDHA, seg_pos={0 0}, upload=57305]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:39.343129266 (59h44m49.065692296s) [noise=true , address=104.36.82.52:4112 , piece_id=4MMFVAPABI2L3TVHLTN5MI4OSVWKHB2PHN2KBZLMRZPTG2OKOS4A, seg_pos={0 0}, upload=57305]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: write tcp 188.166.225.170:49546->104.36.82.52:4112: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:39.343142841 (59h44m49.065705881s) [seg_pos={0 0}, upload=57305]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:20:39.573572615 (59h44m49.296135655s) [upload=57305]: Done uploading segments... Jul 26 23:20:39.573598605 (59h44m49.296161635s) [upload=57305]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.232373462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........................................XXX.X..X.....XXXX] duration: 30.00042934s FAIL (2025-07-26 23:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000440281s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.X........X.X...........................XXXXXXXXXXXXXXXX] duration: 30.239781567s FAIL (2025-07-26 23:20:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.239805282s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...........X.........X...........X.......XXXXXXXXXXXXXXXXXXX] duration: 30.231817771s FAIL (2025-07-26 23:20:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 13255 (59h44m2.260176285s) [seg_pos={0 0}, upload=57310]: Done begin upload segment. Jul 26 23:19:52.537634176 (59h44m2.260197216s) [noise=true , address=45.91.116.13:4153 , piece_id=YLNVY3RXPTZTWG3L3PALZY7YTSJZKRLEQP46LFYTPXN2PN2KZNLA, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.537789131 (59h44m2.260352171s) [noise=true , address=104.36.82.8:4971 , piece_id=3EB7GD2WQ52HFTGSJQHNPIH23AXGDG73ZW5ZV5TBTPJEYQTAUSKQ, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.538027488 (59h44m2.260590528s) [noise=true , address=104.36.82.2:4729 , piece_id=I65YUIYT37KZ63G3IPFPP55ODB4AAUG6K45BTKAS6WHSXCAYPCUA, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.538393508 (59h44m2.260956548s) [noise=true , address=104.36.82.5:4839 , piece_id=3JXDLVRKFAJ4QVPH5FCEH4BYDX7QPZDFNOOEPS64YZKNGAZRBTSA, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.538493880 (59h44m2.26105692s) [noise=true , address=104.36.82.9:4005 , piece_id=MBFBKUAJLSYHSJISHEDKM7TW24HYHOKJGEGHG5XZFNXD3RPUQJJQ, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.538695856 (59h44m2.261258896s) [noise=true , address=104.36.82.13:4186 , piece_id=CITRLXYSAWR574CE75R34LNLJOOYAQ46WU2TFIBTGWXKD52NUOFQ, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.538790237 (59h44m2.261353277s) [noise=true , address=77.90.31.14:4663 , piece_id=R6W53WSKTN6SE77O4MDGJJCGFMZXO667LJVV6AWCLF43KLJSFLTA, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.539027050 (59h44m2.26159009s) [noise=true , address=45.91.116.10:4083 , piece_id=WQDOUEGIV6RFARK7EBJPQDAAIRT46LSOTG37KOSL726QYA32U2OA, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.539031489 (59h44m2.261594529s) [noise=true , address=104.36.82.6:4873 , piece_id=2VBDJSH4SZTMVF7M7HYHHNR2KZ77JO7UXLCNZZHCSUH5JHYNMGUQ, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.539277219 (59h44m2.261840259s) [noise=true , address=77.90.31.21:4877 , piece_id=2DUPQX34MWSGNX4BBT7APKVIC5TCBFYLF3EIKAEQ7UTM4SNYMSUQ, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.539325782 (59h44m2.261888822s) [noise=true , address=104.36.82.9:4007 , piece_id=SP5LJGUYMJ2SJTGTQAPVPDJSG5JGHXRUA2UAEXCG6Z44YMN6HPTA, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.539514984 (59h44m2.262078024s) [noise=true , address=45.91.116.4:4937 , piece_id=VPRUS3FJTTLJ2PQXZQTYUIH6IZTQDPZHRAT4PZ4NE45CAMDUMLBQ, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.539632700 (59h44m2.26219574s) [noise=true , address=104.36.82.15:4258 , piece_id=MVP2NPZMJCAZWBWH4A2TY5YV5JR7B5L6TFZU7C7OJI74JCAQ3CKA, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.539803226 (59h44m2.262366266s) [noise=true , address=45.91.116.14:4189 , piece_id=3QDTOKAVLTWWRS6EI5BRTS2RQ2OWFUNASGZ5KPY64C6NKI76U3VQ, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.539919458 (59h44m2.262482498s) [noise=true , address=104.36.82.7:4911 , piece_id=YP5JPJPLLKXIBYDBKGZXGNTPTLIYVH43ZF2FBXJEPTD6GVDAXB4A, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:52.540054576 (59h44m2.262617617s) [noise=true , address=104.36.82.10:4079 , piece_id=2CJRCQMD5BTBAZZRZ5BTO5ED3EK27FVFMQX7ZUQ6WLT5ABP27EBQ, seg_pos={0 0}, upload=57310]: Uploading piece... Jul 26 23:19:54.283002729 (59h44m4.005565769s) [upload=57310]: Next returned nil segment Jul 26 23:19:54.283019060 (59h44m4.0055821s) [upload=57310]: Waiting for error group managing segments... Jul 26 23:19:54.573225454 (59h44m4.295788484s) [noise=true , address=45.91.116.10:4083 , piece_id=WQDOUEGIV6RFARK7EBJPQDAAIRT46LSOTG37KOSL726QYA32U2OA, seg_pos={0 0}, upload=57310]: Done uploading piece. err: <nil> Jul 26 23:19:54.600423091 (59h44m4.322986131s) [noise=true , address=77.90.31.21:4877 , piece_id=2DUPQX34MWSGNX4BBT7APKVIC5TCBFYLF3EIKAEQ7UTM4SNYMSUQ, seg_pos={0 0}, upload=57310]: Done uploading piece. err: <nil> Jul 26 23:19:54.649602616 (59h44m4.372165656s) [noise=true , address=45.91.116.4:4937 , piece_id=VPRUS3FJTTLJ2PQXZQTYUIH6IZTQDPZHRAT4PZ4NE45CAMDUMLBQ, seg_pos={0 0}, upload=57310]: Done uploading piece. err: <nil> Jul 26 23:19:54.702970465 (59h44m4.425533495s) [noise=true , address=77.90.31.14:4663 , piece_id=R6W53WSKTN6SE77O4MDGJJCGFMZXO667LJVV6AWCLF43KLJSFLTA, seg_pos={0 0}, upload=57310]: Done uploading piece. err: <nil> Jul 26 23:19:54.791514417 (59h44m4.514077457s) [noise=true , address=45.91.116.14:4189 , piece_id=3QDTOKAVLTWWRS6EI5BRTS2RQ2OWFUNASGZ5KPY64C6NKI76U3VQ, seg_pos={0 0}, upload=57310]: Done uploading piece. err: <nil> Jul 26 23:19:54.791986710 (59h44m4.51454974s) [noise=true , address=45.91.116.13:4153 , piece_id=YLNVY3RXPTZTWG3L3PALZY7YTSJZKRLEQP46LFYTPXN2PN2KZNLA, seg_pos={0 0}, upload=57310]: Done uploading piece. err: <nil> Jul 26 23:20:21.660547163 (59h44m31.383110203s) [noise=true , address=104.36.82.9:4005 , piece_id=MBFBKUAJLSYHSJISHEDKM7TW24HYHOKJGEGHG5XZFNXD3RPUQJJQ, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:36662->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660580607 (59h44m31.383143647s) [noise=true , address=104.36.82.5:4839 , piece_id=3JXDLVRKFAJ4QVPH5FCEH4BYDX7QPZDFNOOEPS64YZKNGAZRBTSA, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:54544->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660616987 (59h44m31.383180027s) [noise=true , address=104.36.82.6:4873 , piece_id=2VBDJSH4SZTMVF7M7HYHHNR2KZ77JO7UXLCNZZHCSUH5JHYNMGUQ, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 188.166.225.170:56328->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660630793 (59h44m31.383193833s) [noise=true , address=104.36.82.15:4258 , piece_id=MVP2NPZMJCAZWBWH4A2TY5YV5JR7B5L6TFZU7C7OJI74JCAQ3CKA, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41430->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660666521 (59h44m31.383229561s) [noise=true , address=104.36.82.13:4186 , piece_id=CITRLXYSAWR574CE75R34LNLJOOYAQ46WU2TFIBTGWXKD52NUOFQ, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:57412->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660689545 (59h44m31.383252575s) [noise=true , address=104.36.82.7:4911 , piece_id=YP5JPJPLLKXIBYDBKGZXGNTPTLIYVH43ZF2FBXJEPTD6GVDAXB4A, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660704825 (59h44m31.383267865s) [noise=true , address=104.36.82.9:4007 , piece_id=SP5LJGUYMJ2SJTGTQAPVPDJSG5JGHXRUA2UAEXCG6Z44YMN6HPTA, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:40128->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660735774 (59h44m31.383298814s) [noise=true , address=104.36.82.8:4971 , piece_id=3EB7GD2WQ52HFTGSJQHNPIH23AXGDG73ZW5ZV5TBTPJEYQTAUSKQ, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:55254->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660756674 (59h44m31.383319714s) [noise=true , address=104.36.82.10:4079 , piece_id=2CJRCQMD5BTBAZZRZ5BTO5ED3EK27FVFMQX7ZUQ6WLT5ABP27EBQ, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:37378->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660767164 (59h44m31.383330204s) [noise=true , address=104.36.82.2:4729 , piece_id=I65YUIYT37KZ63G3IPFPP55ODB4AAUG6K45BTKAS6WHSXCAYPCUA, seg_pos={0 0}, upload=57310]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:37892->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:21.660783495 (59h44m31.383346535s) [seg_pos={0 0}, upload=57310]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:20:21.890353952 (59h44m31.612916992s) [upload=57310]: Done uploading segments... Jul 26 23:20:21.890378650 (59h44m31.61294169s) [upload=57310]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.231828851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.841686848s FAIL (2025-07-26 23:20:10 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.841719921s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................XXX.X..X.....XXX] duration: 30.003096676s FAIL (2025-07-26 23:20:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.003112066s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [............X.........X...........X.......XXXXXXXXXXXXXXXXXX] duration: 30.232062872s FAIL (2025-07-26 23:19:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== Begin upload segment... Jul 26 23:19:22.081452487 (59h43m31.804015527s) [seg_pos={0 0}, upload=57301]: Done begin upload segment. Jul 26 23:19:22.081486912 (59h43m31.804049952s) [noise=true , address=104.36.82.7:4911 , piece_id=NWRC3P5HIL46JB4R35UZVXIMSWKKFSP7TDTWNBE5OB3RMSNTD4LQ, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.081510828 (59h43m31.804073858s) [noise=true , address=104.36.82.11:4115 , piece_id=RUN5SFBGFKSO2BNFL6HF4OHBBJ5PWGWIX2P5V6LLHB3OB65C3E7Q, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.081951681 (59h43m31.804514722s) [noise=true , address=104.36.82.11:4114 , piece_id=736BWISUQI6NBVRUOVUHJDMRBILPSTWOBOAJBSWZZWZI2LDPVCPA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.081978783 (59h43m31.804541823s) [noise=true , address=45.91.116.6:4974 , piece_id=72ITRPI5VY6EOPBLCD2XNAIIWLHAD54VMC72SJ76IMX7OOJ6UUBA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.082257276 (59h43m31.804820316s) [noise=true , address=104.36.82.6:4876 , piece_id=H5UIMOLGG7AGQQP64SW4VZK7LLWPSQCZMOZZBMILFSURBBGAGTGA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.082365383 (59h43m31.804928423s) [noise=true , address=104.36.82.4:4802 , piece_id=4FDND5G3K6RIIQAYEPZKCXWIMGFH5QHQOXM35NOPWFFBAO4XJ4SA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.082512404 (59h43m31.805075444s) [noise=true , address=104.36.82.8:4971 , piece_id=JU6IXRB35SFJYWB3TWOWAH22K4BKSCBRKAA2M6MUB23S3DJVBJ5A, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.082706686 (59h43m31.805269726s) [noise=true , address=77.90.31.14:4663 , piece_id=SQUG6QQODOC6MQCZQCDRZKEYL4XGJQGXVLPWR64STPZGH3J4DGHA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.083037428 (59h43m31.805600468s) [noise=true , address=45.91.116.12:4226 , piece_id=GJJAQFBTEETICOQ2QIF5VYLT4M5ZZWMCATKCGRL34LSHQ3TS7RWA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.083383711 (59h43m31.805946751s) [noise=true , address=45.91.116.4:4937 , piece_id=PTV4LE43PM5S2Z6N4QINJ2S4WXAE24ETA5F7BP6Y4QJYPUITSPFA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.083666412 (59h43m31.806229452s) [noise=true , address=104.36.82.2:4731 , piece_id=MHQWFDDUMJ7DGHJZLXUIGHWMHF3A56HM2TZBKA6J4WVYCMVMMNAQ, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.083996163 (59h43m31.806559203s) [noise=true , address=45.91.116.7:4011 , piece_id=LUQQI73OZIL5HM6662MLTRBP6CA6PJHF5ZWBWYEH3WJ563NJZZPA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.084394225 (59h43m31.806957265s) [noise=true , address=104.36.82.16:4295 , piece_id=ZJGY75CZYUTYYBP5YLJPWM64N7VKB2NP2D4R3BKOJSH5OYB673LQ, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.084674661 (59h43m31.807237701s) [noise=true , address=104.36.82.9:4005 , piece_id=OT5GFKKAAPN3GPDJOB36EY4UJZVQC2RXZ2BTCDOGLBDJBENUXTNA, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.084959526 (59h43m31.807522566s) [noise=true , address=104.36.82.3:4765 , piece_id=NAM52JMAGN3OADKWJ7YPWJF3PHXC4FPSRGPS6T4LELDSM3ZKQX5A, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:22.084993841 (59h43m31.807556881s) [noise=true , address=104.36.82.13:4185 , piece_id=6XFTYU743EDA3VAY3SORUH4EHX5Y4ZBDJIXPQ5PQFAMZHDOE26IQ, seg_pos={0 0}, upload=57301]: Uploading piece... Jul 26 23:19:23.881563656 (59h43m33.604126686s) [upload=57301]: Next returned nil segment Jul 26 23:19:23.881586348 (59h43m33.604149378s) [upload=57301]: Waiting for error group managing segments... Jul 26 23:19:24.179529325 (59h43m33.902092365s) [noise=true , address=77.90.31.14:4663 , piece_id=SQUG6QQODOC6MQCZQCDRZKEYL4XGJQGXVLPWR64STPZGH3J4DGHA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: <nil> Jul 26 23:19:24.186811607 (59h43m33.909374637s) [noise=true , address=45.91.116.6:4974 , piece_id=72ITRPI5VY6EOPBLCD2XNAIIWLHAD54VMC72SJ76IMX7OOJ6UUBA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: <nil> Jul 26 23:19:24.203845068 (59h43m33.926408108s) [noise=true , address=45.91.116.7:4011 , piece_id=LUQQI73OZIL5HM6662MLTRBP6CA6PJHF5ZWBWYEH3WJ563NJZZPA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: <nil> Jul 26 23:19:24.207930885 (59h43m33.930493925s) [noise=true , address=45.91.116.4:4937 , piece_id=PTV4LE43PM5S2Z6N4QINJ2S4WXAE24ETA5F7BP6Y4QJYPUITSPFA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: <nil> Jul 26 23:19:24.282385890 (59h43m34.00494892s) [noise=true , address=45.91.116.12:4226 , piece_id=GJJAQFBTEETICOQ2QIF5VYLT4M5ZZWMCATKCGRL34LSHQ3TS7RWA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: <nil> Jul 26 23:19:51.198981925 (59h44m0.921544956s) [noise=true , address=104.36.82.4:4802 , piece_id=4FDND5G3K6RIIQAYEPZKCXWIMGFH5QHQOXM35NOPWFFBAO4XJ4SA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199036349 (59h44m0.921599389s) [noise=true , address=104.36.82.7:4911 , piece_id=NWRC3P5HIL46JB4R35UZVXIMSWKKFSP7TDTWNBE5OB3RMSNTD4LQ, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:52000->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199085193 (59h44m0.921648233s) [noise=true , address=104.36.82.8:4971 , piece_id=JU6IXRB35SFJYWB3TWOWAH22K4BKSCBRKAA2M6MUB23S3DJVBJ5A, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199120131 (59h44m0.921683171s) [noise=true , address=104.36.82.2:4731 , piece_id=MHQWFDDUMJ7DGHJZLXUIGHWMHF3A56HM2TZBKA6J4WVYCMVMMNAQ, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199155879 (59h44m0.921718919s) [noise=true , address=104.36.82.11:4115 , piece_id=RUN5SFBGFKSO2BNFL6HF4OHBBJ5PWGWIX2P5V6LLHB3OB65C3E7Q, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199167350 (59h44m0.92173039s) [noise=true , address=104.36.82.11:4114 , piece_id=736BWISUQI6NBVRUOVUHJDMRBILPSTWOBOAJBSWZZWZI2LDPVCPA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199208680 (59h44m0.92177171s) [noise=true , address=104.36.82.16:4295 , piece_id=ZJGY75CZYUTYYBP5YLJPWM64N7VKB2NP2D4R3BKOJSH5OYB673LQ, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199217476 (59h44m0.921780516s) [noise=true , address=104.36.82.3:4765 , piece_id=NAM52JMAGN3OADKWJ7YPWJF3PHXC4FPSRGPS6T4LELDSM3ZKQX5A, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199254938 (59h44m0.921817978s) [noise=true , address=104.36.82.9:4005 , piece_id=OT5GFKKAAPN3GPDJOB36EY4UJZVQC2RXZ2BTCDOGLBDJBENUXTNA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199270778 (59h44m0.921833818s) [noise=true , address=104.36.82.13:4185 , piece_id=6XFTYU743EDA3VAY3SORUH4EHX5Y4ZBDJIXPQ5PQFAMZHDOE26IQ, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199289815 (59h44m0.921852855s) [noise=true , address=104.36.82.6:4876 , piece_id=H5UIMOLGG7AGQQP64SW4VZK7LLWPSQCZMOZZBMILFSURBBGAGTGA, seg_pos={0 0}, upload=57301]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:51282->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:51.199321486 (59h44m0.921884526s) [seg_pos={0 0}, upload=57301]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:19:51.429102817 (59h44m1.151665857s) [upload=57301]: Done uploading segments... Jul 26 23:19:51.429129308 (59h44m1.151692348s) [upload=57301]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.232088321s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.874676708s FAIL (2025-07-26 23:19:43 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.874705522s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...........X..XX...X.X.X.X.......XX..XX.XX..XXX............X] duration: 1m0.000963985s FAIL (2025-07-26 23:19:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000985817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.239742671s FAIL (2025-07-26 23:19:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== s) [upload=57293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:18:39.506578364 (59h42m49.229141404s) [upload=57293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:18:39.955816182 (59h42m49.678379222s) [seg_pos={0 0}, upload=57293]: Begin upload segment... Jul 26 23:18:39.955855958 (59h42m49.678418998s) [seg_pos={0 0}, upload=57293]: Done begin upload segment. Jul 26 23:18:39.955875104 (59h42m49.678438144s) [noise=true , address=104.36.82.55:4196 , piece_id=UJIF65OMCNKDAZ2RQBZAZC6LWKEFXH7DI34KQZQYPGROU5ZXLZTA, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.956468520 (59h42m49.67903156s) [noise=true , address=77.90.31.11:4573 , piece_id=E4QVTPW63VGMIZOC7MLKTNHKIOCP3YYGY3QR7ILQQB4RQGMYSB4A, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.956749267 (59h42m49.679312297s) [noise=true , address=77.90.31.17:4799 , piece_id=XPIIMEJRUPWI52OEHWJSXNJYI4KZMBTHUKWPEGYG262E2BODBTNQ, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.957005899 (59h42m49.679568939s) [noise=true , address=104.36.82.46:4508 , piece_id=GMISHD77PIEDLIRWQTYDF5EUOOFZ76ZE2AIV7CZBP62N2Y3O2AYQ, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.957271727 (59h42m49.679834767s) [noise=true , address=79.127.205.234:4610 , piece_id=4O2IWIO56CKZ447ALIUPDRET4JKO3ZRYYVVTU443QNLBWDNWFTYQ, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.957496146 (59h42m49.680059186s) [noise=true , address=77.90.31.10:4531 , piece_id=XQF25HMM5IKZYJHS233MGVBX3WX2SBSBFGT75THWQHXDWD7Z52DA, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.957697071 (59h42m49.680260111s) [noise=true , address=79.127.205.245:4218 , piece_id=DQZRYPW2VBXWVUFPNNUIJCA43FFTJAW7LQBZZJDVUM5YAIVUANUA, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.957986294 (59h42m49.680549334s) [noise=true , address=77.90.31.21:4907 , piece_id=K6YGQDDONXGGKEIPUPIOPBTZUJ6YLCH26I2WWW5PFBR5ONYENI3A, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.958334060 (59h42m49.6808971s) [noise=true , address=104.36.82.47:4539 , piece_id=HMXUPAJ3SQYHVJ2ULZ6RUSCDF6VPMOJSR3I74S2LRUNPQC6HAQ2A, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.958738414 (59h42m49.681301444s) [noise=true , address=79.127.219.38:4015 , piece_id=MJYQAT6UH4OO3YX6A7YNBJ4Y22GDHWQEWBUWTVAR4ZANMCRE2QLA, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.959000975 (59h42m49.681564015s) [noise=true , address=79.127.219.47:4061 , piece_id=XPGUF5AN2JQHFNWPMOBW3NXNKNGRAMGTPYQFXR7SWTDDT5LVZKKA, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.959296681 (59h42m49.681859711s) [noise=true , address=104.36.82.53:4122 , piece_id=YBKCBBCIO6IH57INNT65PNMNWM3B55YNZRQMSOBQPTJSJA4BKKPQ, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.959591706 (59h42m49.682154746s) [noise=true , address=79.127.201.215:4347 , piece_id=G4GRQS52BXEPIRYX7R7QO4U43JBLDQINXDO5EN4N3PVEV636LTIQ, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.960235678 (59h42m49.682798738s) [noise=true , address=77.90.31.18:4828 , piece_id=H4CBE42RHIDYTX2D2ZGH7R3ZR7UBC7SWAAFJTQMFLL2AFIQKCMXA, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.960525543 (59h42m49.683088583s) [noise=true , address=79.127.220.97:4361 , piece_id=I4SPE6VESFGVZL54WYR2WZ4YDYAOYXDRMKINRM7WE7ZR6IHXE23Q, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.960748479 (59h42m49.683311519s) [noise=true , address=104.36.82.52:4102 , piece_id=X24G7RVUTWKFX6FQTV5E4LLRLKQPD7AQA72GRVZRZH5CBMHEYQUA, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.961001463 (59h42m49.683564503s) [noise=true , address=79.127.205.246:4269 , piece_id=KTFPWTC4KQQKGA6POFQWLMXWZPZY4EKJP4POJ3L5GNORZZ67264Q, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:39.961202668 (59h42m49.683765708s) [noise=true , address=64.49.13.139:4478 , piece_id=WQ5WEH7IEKO2LKOUUXASWSZLIB4PGLKJUMUVPPQDJHB6TEQ2XQOQ, seg_pos={0 0}, upload=57293]: Uploading piece... Jul 26 23:18:41.055558784 (59h42m50.778121824s) [upload=57293]: Next returned nil segment Jul 26 23:18:41.055586156 (59h42m50.778149447s) [upload=57293]: Waiting for error group managing segments... Jul 26 23:18:41.360844508 (59h42m51.083407548s) [noise=true , address=79.127.205.245:4218 , piece_id=DQZRYPW2VBXWVUFPNNUIJCA43FFTJAW7LQBZZJDVUM5YAIVUANUA, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:41.365321444 (59h42m51.087884474s) [noise=true , address=79.127.205.234:4610 , piece_id=4O2IWIO56CKZ447ALIUPDRET4JKO3ZRYYVVTU443QNLBWDNWFTYQ, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:41.366254820 (59h42m51.08881786s) [noise=true , address=79.127.205.246:4269 , piece_id=KTFPWTC4KQQKGA6POFQWLMXWZPZY4EKJP4POJ3L5GNORZZ67264Q, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:41.371790431 (59h42m51.094353471s) [noise=true , address=64.49.13.139:4478 , piece_id=WQ5WEH7IEKO2LKOUUXASWSZLIB4PGLKJUMUVPPQDJHB6TEQ2XQOQ, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:41.853654023 (59h42m51.576217063s) [noise=true , address=77.90.31.11:4573 , piece_id=E4QVTPW63VGMIZOC7MLKTNHKIOCP3YYGY3QR7ILQQB4RQGMYSB4A, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:41.854442963 (59h42m51.577006003s) [noise=true , address=79.127.219.38:4015 , piece_id=MJYQAT6UH4OO3YX6A7YNBJ4Y22GDHWQEWBUWTVAR4ZANMCRE2QLA, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:42.030234215 (59h42m51.752797245s) [noise=true , address=77.90.31.18:4828 , piece_id=H4CBE42RHIDYTX2D2ZGH7R3ZR7UBC7SWAAFJTQMFLL2AFIQKCMXA, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:42.033589024 (59h42m51.756152054s) [noise=true , address=77.90.31.21:4907 , piece_id=K6YGQDDONXGGKEIPUPIOPBTZUJ6YLCH26I2WWW5PFBR5ONYENI3A, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:44.818102356 (59h42m54.540665396s) [noise=true , address=79.127.220.97:4361 , piece_id=I4SPE6VESFGVZL54WYR2WZ4YDYAOYXDRMKINRM7WE7ZR6IHXE23Q, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:18:45.402781795 (59h42m55.125344835s) [noise=true , address=79.127.219.47:4061 , piece_id=XPGUF5AN2JQHFNWPMOBW3NXNKNGRAMGTPYQFXR7SWTDDT5LVZKKA, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:19:06.988898606 (59h43m16.711461646s) [noise=true , address=79.127.201.215:4347 , piece_id=G4GRQS52BXEPIRYX7R7QO4U43JBLDQINXDO5EN4N3PVEV636LTIQ, seg_pos={0 0}, upload=57293]: Done uploading piece. err: <nil> Jul 26 23:19:38.861641061 (59h43m48.584204101s) [noise=true , address=104.36.82.47:4539 , piece_id=HMXUPAJ3SQYHVJ2ULZ6RUSCDF6VPMOJSR3I74S2LRUNPQC6HAQ2A, seg_pos={0 0}, upload=57293]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: rpc: dial tcp 104.36.82.47:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:38.861686988 (59h43m48.584250028s) [noise=true , address=104.36.82.55:4196 , piece_id=UJIF65OMCNKDAZ2RQBZAZC6LWKEFXH7DI34KQZQYPGROU5ZXLZTA, seg_pos={0 0}, upload=57293]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: rpc: dial tcp 104.36.82.55:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:38.861719992 (59h43m48.584283032s) [noise=true , address=77.90.31.17:4799 , piece_id=XPIIMEJRUPWI52OEHWJSXNJYI4KZMBTHUKWPEGYG262E2BODBTNQ, seg_pos={0 0}, upload=57293]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:38.861753686 (59h43m48.584316726s) [noise=true , address=104.36.82.53:4122 , piece_id=YBKCBBCIO6IH57INNT65PNMNWM3B55YNZRQMSOBQPTJSJA4BKKPQ, seg_pos={0 0}, upload=57293]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: rpc: dial tcp 104.36.82.53:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:38.861853578 (59h43m48.584416618s) [noise=true , address=104.36.82.46:4508 , piece_id=GMISHD77PIEDLIRWQTYDF5EUOOFZ76ZE2AIV7CZBP62N2Y3O2AYQ, seg_pos={0 0}, upload=57293]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:19:38.861930715 (59h43m48.584493745s) [noise=true , address=77.90.31.10:4531 , piece_id=XQF25HMM5IKZYJHS233MGVBX3WX2SBSBFGT75THWQHXDWD7Z52DA, seg_pos={0 0}, upload=57293]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:19:38.862155936 (59h43m48.584718976s) [noise=true , address=104.36.82.52:4102 , piece_id=X24G7RVUTWKFX6FQTV5E4LLRLKQPD7AQA72GRVZRZH5CBMHEYQUA, seg_pos={0 0}, upload=57293]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:19:38.862194880 (59h43m48.58475792s) [seg_pos={0 0}, upload=57293]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:19:39.099357465 (59h43m48.821920505s) [upload=57293]: Done uploading segments... Jul 26 23:19:39.099382503 (59h43m48.821945543s) [upload=57293]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.239754662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.878820876s FAIL (2025-07-26 23:19:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.878849902s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X.X........X.X...........................XXXXXXXXXXXXXXX] duration: 30.232946868s FAIL (2025-07-26 23:19:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.232987997s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................................XXX.X..X.....XX] duration: 30.000945318s FAIL (2025-07-26 23:19:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000957702s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.............X.........X...........X.......XXXXXXXXXXXXXXXXX] duration: 30.231469424s FAIL (2025-07-26 23:18:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== FYZX2IJKW7V4LF27E57JI5Q, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637222606 (59h42m36.359785646s) [noise=true , address=104.36.82.10:4078 , piece_id=GOG2ELJV7PASKKLIQL66SZNPDVNI7CS7PZYIKMYJFUE7WL2BHQ3Q, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637582695 (59h42m36.360145735s) [noise=true , address=77.90.31.14:4663 , piece_id=XOI5ZJNMDK7ECJOITJ4FKGYRSUE4K4TVGTHYREUTMQX6IGQBU27Q, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637598786 (59h42m36.360161817s) [noise=true , address=104.36.82.4:4804 , piece_id=SJ2XYJ33TRYMZ76YPEBTHR5BCGXINKFO7TZWYQH2XWNVBMD6YF6A, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637852381 (59h42m36.360415422s) [noise=true , address=104.36.82.6:4874 , piece_id=WNPITA6QHRV36U6FZJCXNE6ZSXHX3MPMPXZOIXQ3RMUR426FOCVQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.637907337 (59h42m36.360470377s) [noise=true , address=104.36.82.5:4838 , piece_id=E253BSYUKXW66GO3VQCYLRE3OMOD5FPS7JMXJ4FF4HPT3KXC5MBQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638112459 (59h42m36.360675489s) [noise=true , address=104.36.82.7:4909 , piece_id=RVI2I5VLI7KN7AO3JO7X6O75XY3LVJU22NIIT52B4Q63QUDPB3TA, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638202291 (59h42m36.360765331s) [noise=true , address=104.36.82.9:4005 , piece_id=Y6YHHK5WVJZPSAH5RK4O55F2FD75VCGLJBK7MRUMGEZQRDRUFLBQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638362627 (59h42m36.360925667s) [noise=true , address=77.90.31.10:4518 , piece_id=UKCAYXKYD6CFUEANJNEA2WVW7GC4E7UYZSZGXYF6QBLMEL54MRKQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638511312 (59h42m36.361074352s) [noise=true , address=104.36.82.14:4223 , piece_id=YYLEHMNMM4VXPL2QS5VBHNDCMQZUG5G3UQF4F6LFBFRIOWYY67NQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638636401 (59h42m36.361199431s) [noise=true , address=45.91.116.7:4009 , piece_id=4GXRRUQYDEZKH55WM75GPGGW72B3BM52JFI23SW3WFFCEMKLGNHQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638814312 (59h42m36.361377362s) [noise=true , address=104.36.82.4:4803 , piece_id=VG5NDUJOFY36VQCWSPKMQ2EGWA73JLQVYS2HHV3IWZ5VXBKP3LGA, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.638904093 (59h42m36.361467123s) [noise=true , address=45.91.116.13:4154 , piece_id=ENPUWANJ4IFFG5UMDJFN7IAQBLQE7IDS6FD5WDIHH575KZHCLRDA, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.639138212 (59h42m36.361701252s) [noise=true , address=45.91.116.5:4298 , piece_id=MUH4L5JURP6CRJYMZV5AXF3ZDX23SD4XMGYXLFTLSKZS536H7POQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.639342332 (59h42m36.361905362s) [noise=true , address=104.36.82.8:4969 , piece_id=6E4DJXDCDCLKE7STRF65PE7GCWYG6HCNCRNBUGWO7VF2MROTT7OQ, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:26.639815247 (59h42m36.362378287s) [noise=true , address=104.36.82.15:4258 , piece_id=SMMNKQOQIDYBTEL5M2XXGQVNWVK3JMA3XI3E6DIFDD4TAM5EQSCA, seg_pos={0 0}, upload=57286]: Uploading piece... Jul 26 23:18:28.329056920 (59h42m38.05161996s) [upload=57286]: Next returned nil segment Jul 26 23:18:28.329072469 (59h42m38.051635509s) [upload=57286]: Waiting for error group managing segments... Jul 26 23:18:28.659986910 (59h42m38.38254995s) [noise=true , address=45.91.116.5:4298 , piece_id=MUH4L5JURP6CRJYMZV5AXF3ZDX23SD4XMGYXLFTLSKZS536H7POQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: <nil> Jul 26 23:18:28.662276450 (59h42m38.38483948s) [noise=true , address=45.91.116.13:4154 , piece_id=ENPUWANJ4IFFG5UMDJFN7IAQBLQE7IDS6FD5WDIHH575KZHCLRDA, seg_pos={0 0}, upload=57286]: Done uploading piece. err: <nil> Jul 26 23:18:28.740573276 (59h42m38.463136316s) [noise=true , address=77.90.31.14:4663 , piece_id=XOI5ZJNMDK7ECJOITJ4FKGYRSUE4K4TVGTHYREUTMQX6IGQBU27Q, seg_pos={0 0}, upload=57286]: Done uploading piece. err: <nil> Jul 26 23:18:28.846458196 (59h42m38.569021236s) [noise=true , address=45.91.116.7:4009 , piece_id=4GXRRUQYDEZKH55WM75GPGGW72B3BM52JFI23SW3WFFCEMKLGNHQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: <nil> Jul 26 23:18:55.758542517 (59h43m5.481105557s) [noise=true , address=104.36.82.15:4258 , piece_id=SMMNKQOQIDYBTEL5M2XXGQVNWVK3JMA3XI3E6DIFDD4TAM5EQSCA, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758587283 (59h43m5.481150313s) [noise=true , address=104.36.82.6:4874 , piece_id=WNPITA6QHRV36U6FZJCXNE6ZSXHX3MPMPXZOIXQ3RMUR426FOCVQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758618432 (59h43m5.481181462s) [noise=true , address=104.36.82.14:4223 , piece_id=YYLEHMNMM4VXPL2QS5VBHNDCMQZUG5G3UQF4F6LFBFRIOWYY67NQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758647979 (59h43m5.481211019s) [noise=true , address=104.36.82.9:4005 , piece_id=Y6YHHK5WVJZPSAH5RK4O55F2FD75VCGLJBK7MRUMGEZQRDRUFLBQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758699417 (59h43m5.481262447s) [noise=true , address=104.36.82.4:4803 , piece_id=VG5NDUJOFY36VQCWSPKMQ2EGWA73JLQVYS2HHV3IWZ5VXBKP3LGA, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758737940 (59h43m5.48130098s) [noise=true , address=104.36.82.8:4969 , piece_id=6E4DJXDCDCLKE7STRF65PE7GCWYG6HCNCRNBUGWO7VF2MROTT7OQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758788257 (59h43m5.481351297s) [noise=true , address=104.36.82.10:4078 , piece_id=GOG2ELJV7PASKKLIQL66SZNPDVNI7CS7PZYIKMYJFUE7WL2BHQ3Q, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758858852 (59h43m5.481421892s) [noise=true , address=104.36.82.13:4185 , piece_id=4ILYMZHGBXFKP3AHC6XEH62VMMVUTFYZX2IJKW7V4LF27E57JI5Q, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:46998->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758894130 (59h43m5.48145717s) [noise=true , address=104.36.82.5:4838 , piece_id=E253BSYUKXW66GO3VQCYLRE3OMOD5FPS7JMXJ4FF4HPT3KXC5MBQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:50440->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758913687 (59h43m5.481476727s) [noise=true , address=104.36.82.7:4909 , piece_id=RVI2I5VLI7KN7AO3JO7X6O75XY3LVJU22NIIT52B4Q63QUDPB3TA, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758926221 (59h43m5.481489261s) [noise=true , address=104.36.82.4:4804 , piece_id=SJ2XYJ33TRYMZ76YPEBTHR5BCGXINKFO7TZWYQH2XWNVBMD6YF6A, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758964525 (59h43m5.481527565s) [noise=true , address=77.90.31.10:4518 , piece_id=UKCAYXKYD6CFUEANJNEA2WVW7GC4E7UYZSZGXYF6QBLMEL54MRKQ, seg_pos={0 0}, upload=57286]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 188.166.225.170:39552->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:55.758977619 (59h43m5.481540659s) [seg_pos={0 0}, upload=57286]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Jul 26 23:18:55.988191826 (59h43m5.710754856s) [upload=57286]: Done uploading segments... Jul 26 23:18:55.988244526 (59h43m5.710807567s) [upload=57286]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.231481339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230501828s FAIL (2025-07-26 23:18:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== address=104.36.82.55:4215 , piece_id=HROJI66WXDCYQBNP7GNWZEVWKNY2XDQHHNGKMERNGTCWIZ7C4S5Q, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.582172539 (59h41m49.304735579s) [noise=true , address=77.90.31.16:4737 , piece_id=53SFWGTX7ZQHZFG5PG2AOCYABZ7CAMSYNHR36UAMWIGY244KUPVA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.582236822 (59h41m49.304799872s) [noise=true , address=79.127.205.233:4655 , piece_id=P3JGPQ3J3RDO2745RRM5BOCVHDO2W7ET66H3ADGHB625VL3ETKKA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.582991426 (59h41m49.305554516s) [noise=true , address=77.90.31.13:4636 , piece_id=MBGEVNEPGQ474FQHFIICLAFWBHRIE2F3IFMZMJTZNA3YC3RJ7KOQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.583472066 (59h41m49.306035116s) [noise=true , address=104.36.82.47:4543 , piece_id=25L4ITRFE3WLY5STEWUYEAGSVCREJOHHGOGMG5Q2GEPAIC6DEY4A, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.583942856 (59h41m49.306505896s) [noise=true , address=104.36.82.19:4402 , piece_id=MNDMEK6QYHSPUO3CTDEO4MQVIYHZU34G5IGYODK6RF5TGGAGCLOQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.584403968 (59h41m49.306967018s) [noise=true , address=104.36.82.46:4513 , piece_id=5RVJHBCSVJRRVFWDELFEBP746NCGFSVNSACCZNJX3SJGXAT2TNWA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.584782613 (59h41m49.307345653s) [noise=true , address=77.90.31.11:4579 , piece_id=FPIZFOWNQWOT4WB5C4PHQEZWJID4HT2M4EQFTXZR25GEDRSRPERA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.585152340 (59h41m49.307715381s) [noise=true , address=104.36.82.52:4099 , piece_id=FHW74NJHGJO3PDCVOJFJ3FHXADVLTN4OKBCGPRPFACSLRGGRUILQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.585440752 (59h41m49.308003792s) [noise=true , address=104.36.82.53:4141 , piece_id=VIW33FWGS52QA5PNTCVA7EG4G3NYHOZP4O3PNZIVMAYH6F5AW72Q, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.585752478 (59h41m49.308315518s) [noise=true , address=79.127.205.245:4249 , piece_id=UCWUBRUFRK5JS3IIHI6YYRFORSIHZSDODZC3DAFPDW4W6EU36OAA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.586148456 (59h41m49.308711496s) [noise=true , address=79.127.205.234:4615 , piece_id=WFCANOT4STSC3F4BHSOUROVL6QSLVSXEH2VQIF32GCFXZUAOGHNQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.586427981 (59h41m49.308991021s) [noise=true , address=64.49.13.140:4514 , piece_id=QRJJYRYMNQSIVIRGCYU7WC3T7RKGTPQL5VKY4UASK6YAZYMLDNCQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.586699861 (59h41m49.309262901s) [noise=true , address=64.49.13.138:4482 , piece_id=RLB35JTBM3UTZWWPZC7WLK53LDR3TOFYSVGPGEK3UYRR2KHDMBLA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.587004033 (59h41m49.309567073s) [noise=true , address=104.36.82.54:4185 , piece_id=JVS64522A35LTOMIBRP4VPOE2F24KBYC2LAXKVX3BVK54NNDKW7A, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.587296303 (59h41m49.309859343s) [noise=true , address=77.90.31.18:4828 , piece_id=5CT6S2LHXUECI3CVKFLJGJLDQYHQLJOBK4WCMIDUSEZ6KW3CKRSQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.587557341 (59h41m49.310120382s) [noise=true , address=79.127.219.38:4004 , piece_id=TTZFRFMYKQUNE3JTLEKG75D47GV3AKR4ASIRX56SXMZXFKTHXDYQ, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:39.588280105 (59h41m49.310843155s) [noise=true , address=104.36.82.56:4259 , piece_id=LZAQU5TFCKAP6GO2MMRB3BHB7VDSFMLBO5WMQF3XTWIPKPV5H7VA, seg_pos={0 0}, upload=57272]: Uploading piece... Jul 26 23:17:40.697095606 (59h41m50.419658646s) [upload=57272]: Next returned nil segment Jul 26 23:17:40.697110614 (59h41m50.419673654s) [upload=57272]: Waiting for error group managing segments... Jul 26 23:17:40.850969162 (59h41m50.573532202s) [noise=true , address=64.49.13.138:4482 , piece_id=RLB35JTBM3UTZWWPZC7WLK53LDR3TOFYSVGPGEK3UYRR2KHDMBLA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:17:40.983070762 (59h41m50.705633802s) [noise=true , address=79.127.205.234:4615 , piece_id=WFCANOT4STSC3F4BHSOUROVL6QSLVSXEH2VQIF32GCFXZUAOGHNQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.035315874 (59h41m50.757878914s) [noise=true , address=79.127.205.245:4249 , piece_id=UCWUBRUFRK5JS3IIHI6YYRFORSIHZSDODZC3DAFPDW4W6EU36OAA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.248241625 (59h41m50.970804665s) [noise=true , address=79.127.205.233:4655 , piece_id=P3JGPQ3J3RDO2745RRM5BOCVHDO2W7ET66H3ADGHB625VL3ETKKA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.466347097 (59h41m51.188910137s) [noise=true , address=77.90.31.18:4828 , piece_id=5CT6S2LHXUECI3CVKFLJGJLDQYHQLJOBK4WCMIDUSEZ6KW3CKRSQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.495291166 (59h41m51.217854206s) [noise=true , address=79.127.219.38:4004 , piece_id=TTZFRFMYKQUNE3JTLEKG75D47GV3AKR4ASIRX56SXMZXFKTHXDYQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.537493632 (59h41m51.260056672s) [noise=true , address=77.90.31.16:4737 , piece_id=53SFWGTX7ZQHZFG5PG2AOCYABZ7CAMSYNHR36UAMWIGY244KUPVA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.545015775 (59h41m51.267578815s) [noise=true , address=104.36.82.54:4185 , piece_id=JVS64522A35LTOMIBRP4VPOE2F24KBYC2LAXKVX3BVK54NNDKW7A, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:41.594943411 (59h41m51.317506441s) [noise=true , address=77.90.31.11:4579 , piece_id=FPIZFOWNQWOT4WB5C4PHQEZWJID4HT2M4EQFTXZR25GEDRSRPERA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:17:42.524604555 (59h41m52.247167595s) [noise=true , address=64.49.13.140:4514 , piece_id=QRJJYRYMNQSIVIRGCYU7WC3T7RKGTPQL5VKY4UASK6YAZYMLDNCQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:18:02.277063189 (59h42m11.99962626s) [noise=true , address=77.90.31.13:4636 , piece_id=MBGEVNEPGQ474FQHFIICLAFWBHRIE2F3IFMZMJTZNA3YC3RJ7KOQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: <nil> Jul 26 23:18:38.424884073 (59h42m48.147447113s) [noise=true , address=104.36.82.46:4513 , piece_id=5RVJHBCSVJRRVFWDELFEBP746NCGFSVNSACCZNJX3SJGXAT2TNWA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: rpc: dial tcp 104.36.82.46:4513: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.424929109 (59h42m48.147492149s) [noise=true , address=104.36.82.47:4543 , piece_id=25L4ITRFE3WLY5STEWUYEAGSVCREJOHHGOGMG5Q2GEPAIC6DEY4A, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: rpc: dial tcp 104.36.82.47:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.424942846 (59h42m48.147505886s) [noise=true , address=104.36.82.55:4215 , piece_id=HROJI66WXDCYQBNP7GNWZEVWKNY2XDQHHNGKMERNGTCWIZ7C4S5Q, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 188.166.225.170:41890->104.36.82.55:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.424966892 (59h42m48.147529932s) [noise=true , address=104.36.82.56:4259 , piece_id=LZAQU5TFCKAP6GO2MMRB3BHB7VDSFMLBO5WMQF3XTWIPKPV5H7VA, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: write tcp 188.166.225.170:49700->104.36.82.56:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.424996037 (59h42m48.147559078s) [noise=true , address=104.36.82.19:4402 , piece_id=MNDMEK6QYHSPUO3CTDEO4MQVIYHZU34G5IGYODK6RF5TGGAGCLOQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:104.36.82.19:4402): protocol: rpc: dial tcp 104.36.82.19:4402: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.425032477 (59h42m48.147595517s) [noise=true , address=104.36.82.52:4099 , piece_id=FHW74NJHGJO3PDCVOJFJ3FHXADVLTN4OKBCGPRPFACSLRGGRUILQ, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT, address:104.36.82.52:4099): protocol: rpc: dial tcp 104.36.82.52:4099: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.425061733 (59h42m48.147624773s) [noise=true , address=104.36.82.53:4141 , piece_id=VIW33FWGS52QA5PNTCVA7EG4G3NYHOZP4O3PNZIVMAYH6F5AW72Q, seg_pos={0 0}, upload=57272]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: write tcp 188.166.225.170:55394->104.36.82.53:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:38.425080128 (59h42m48.147643168s) [seg_pos={0 0}, upload=57272]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:18:38.653644601 (59h42m48.376207641s) [upload=57272]: Done uploading segments... Jul 26 23:18:38.653685079 (59h42m48.376248109s) [upload=57272]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230517077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [............X..XX...X.X.X.X.......XX..XX.XX..XXX............] duration: 19.430575873s FAIL (2025-07-26 23:18:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.43059578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..............X.........X...........X.......XXXXXXXXXXXXXXXX] duration: 30.236722725s FAIL (2025-07-26 23:18:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== east 13 but got 9) Jul 26 23:17:55.945461463 (59h42m5.668024493s) [upload=57278]: Starting upload Jul 26 23:17:55.945518392 (59h42m5.668081432s) [upload=57278]: Done starting upload Jul 26 23:17:55.945571363 (59h42m5.668134403s) [upload=57278]: Uploading segments... Jul 26 23:17:55.945689339 (59h42m5.668252379s) [upload=57278]: Got next segment. Inline: false Jul 26 23:17:55.945715268 (59h42m5.668278308s) [upload=57278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:17:55.945719506 (59h42m5.668282546s) [upload=57278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:17:56.180132761 (59h42m5.902695801s) [seg_pos={0 0}, upload=57278]: Begin upload segment... Jul 26 23:17:56.180171965 (59h42m5.902735005s) [seg_pos={0 0}, upload=57278]: Done begin upload segment. Jul 26 23:17:56.180239425 (59h42m5.902802465s) [noise=true , address=104.36.82.3:4767 , piece_id=4VYMQPS34INP5Q2PYW6FO2DGP4Z6ZSF6BIERUU26DQFDE2X5ZBCQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.180285372 (59h42m5.902848412s) [noise=true , address=45.91.116.6:4975 , piece_id=FMBIGDRVIX4363YSFCGHU5B74LWT7Y63E3UAHBPGQG3RTU4YAVBQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.180729111 (59h42m5.903292151s) [noise=true , address=104.36.82.10:4079 , piece_id=2JGXXC7DW3PFF3WZU7CU7S4MXAJ4ICMHSZJ4HKY33XC7EEPLLQEA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.180755902 (59h42m5.903318942s) [noise=true , address=45.91.116.7:4010 , piece_id=P5BQ4ECAVUJO445S2XQ4AISX6VYFIJ42QHKICROMPFEZNQU7J4GA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181014929 (59h42m5.903577979s) [noise=true , address=104.36.82.11:4115 , piece_id=T75TOGWKL3PYYSBQBMTYJZCOKJBIIGHYUZFEUTWXJWAMSY4PYSFA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181087889 (59h42m5.903650929s) [noise=true , address=104.36.82.13:4185 , piece_id=RJZXED5DVMIC7RQMBZX6THKHNYPQUXHHOD4GNOTYEETGGQ2RTQMA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181283713 (59h42m5.903846753s) [noise=true , address=45.91.116.10:4082 , piece_id=XFKFSCMKDL2S5TDTXH5MUL2UCP3BG56ISKIQDAHK64ZAWL2XNWBQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181370559 (59h42m5.903933599s) [noise=true , address=104.36.82.7:4910 , piece_id=M2VYAEIAYPWTKC6UHNFFPR4SOSPBUUUN7KWYXBMHLZURD4EVVH3Q, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181529814 (59h42m5.904092844s) [noise=true , address=104.36.82.15:4257 , piece_id=UA3HB6N2HXPKQIBYZO7B2HMTP5X426VHT65MPAKHHDU7P3IVZJRA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181667297 (59h42m5.904230337s) [noise=true , address=45.91.116.4:4937 , piece_id=5NW5P4JPWXD3OIWOOOUF63JMRNUVZZMLS2TE4IBJEU75CFJMYKAA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.181786865 (59h42m5.904349906s) [noise=true , address=77.90.31.17:4770 , piece_id=HZB2R73FJCWPNQSARFBARD2MZXIO2VRAYGU4OHJ7IMJRURJ2UGIA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.182067332 (59h42m5.904630362s) [noise=true , address=45.91.116.14:4191 , piece_id=XJCD6V2UBXDCJCQVHGCOVLWG6YTAK3RDUM33PJILYH2E3KJT4DEA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.182261634 (59h42m5.904824674s) [noise=true , address=104.36.82.13:4187 , piece_id=RKCOQPWQKJAIA6V3MWOS4WGGFRZ7YAXGCPWGJBZ7AKDW6ECMC7OQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.182455705 (59h42m5.905018745s) [noise=true , address=45.91.116.6:4973 , piece_id=PKO7CGBRCKQEEA2QB5SPSDKEG55X3JXWLYWJQCZSWRJ5H6CSNBDQ, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.183891642 (59h42m5.906454682s) [noise=true , address=104.36.82.2:4730 , piece_id=MUMTIPJSM2X2Q73IGPRXNU774EXZ7LZQ4AMB3IHRA2OT4NBCEJHA, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:56.184137723 (59h42m5.906700753s) [noise=true , address=104.36.82.10:4077 , piece_id=4UF2CDXZFI7LF2QR6AGQUCFDOWZLCU26J4RWRIUSQ4BLHEFUPV3A, seg_pos={0 0}, upload=57278]: Uploading piece... Jul 26 23:17:57.984487703 (59h42m7.707050743s) [upload=57278]: Next returned nil segment Jul 26 23:17:57.984502241 (59h42m7.707065271s) [upload=57278]: Waiting for error group managing segments... Jul 26 23:17:58.280634973 (59h42m8.003198003s) [noise=true , address=45.91.116.6:4973 , piece_id=PKO7CGBRCKQEEA2QB5SPSDKEG55X3JXWLYWJQCZSWRJ5H6CSNBDQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.293320285 (59h42m8.015883325s) [noise=true , address=45.91.116.10:4082 , piece_id=XFKFSCMKDL2S5TDTXH5MUL2UCP3BG56ISKIQDAHK64ZAWL2XNWBQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.300193614 (59h42m8.022756654s) [noise=true , address=45.91.116.7:4010 , piece_id=P5BQ4ECAVUJO445S2XQ4AISX6VYFIJ42QHKICROMPFEZNQU7J4GA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.300569674 (59h42m8.023132704s) [noise=true , address=45.91.116.6:4975 , piece_id=FMBIGDRVIX4363YSFCGHU5B74LWT7Y63E3UAHBPGQG3RTU4YAVBQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.364854361 (59h42m8.087417401s) [noise=true , address=45.91.116.4:4937 , piece_id=5NW5P4JPWXD3OIWOOOUF63JMRNUVZZMLS2TE4IBJEU75CFJMYKAA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:17:58.429865256 (59h42m8.152428296s) [noise=true , address=45.91.116.14:4191 , piece_id=XJCD6V2UBXDCJCQVHGCOVLWG6YTAK3RDUM33PJILYH2E3KJT4DEA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:18:14.206751585 (59h42m23.929314675s) [noise=true , address=104.36.82.13:4187 , piece_id=RKCOQPWQKJAIA6V3MWOS4WGGFRZ7YAXGCPWGJBZ7AKDW6ECMC7OQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:18:14.321264263 (59h42m24.043827303s) [noise=true , address=104.36.82.13:4185 , piece_id=RJZXED5DVMIC7RQMBZX6THKHNYPQUXHHOD4GNOTYEETGGQ2RTQMA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: <nil> Jul 26 23:18:25.296932302 (59h42m35.019495342s) [noise=true , address=104.36.82.7:4910 , piece_id=M2VYAEIAYPWTKC6UHNFFPR4SOSPBUUUN7KWYXBMHLZURD4EVVH3Q, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.296977869 (59h42m35.019540909s) [noise=true , address=104.36.82.11:4115 , piece_id=T75TOGWKL3PYYSBQBMTYJZCOKJBIIGHYUZFEUTWXJWAMSY4PYSFA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297010061 (59h42m35.019573101s) [noise=true , address=104.36.82.15:4257 , piece_id=UA3HB6N2HXPKQIBYZO7B2HMTP5X426VHT65MPAKHHDU7P3IVZJRA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297046590 (59h42m35.01960963s) [noise=true , address=104.36.82.2:4730 , piece_id=MUMTIPJSM2X2Q73IGPRXNU774EXZ7LZQ4AMB3IHRA2OT4NBCEJHA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:58774->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297209792 (59h42m35.019772842s) [noise=true , address=77.90.31.17:4770 , piece_id=HZB2R73FJCWPNQSARFBARD2MZXIO2VRAYGU4OHJ7IMJRURJ2UGIA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297253296 (59h42m35.019816336s) [noise=true , address=104.36.82.10:4077 , piece_id=4UF2CDXZFI7LF2QR6AGQUCFDOWZLCU26J4RWRIUSQ4BLHEFUPV3A, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297290978 (59h42m35.019854018s) [noise=true , address=104.36.82.3:4767 , piece_id=4VYMQPS34INP5Q2PYW6FO2DGP4Z6ZSF6BIERUU26DQFDE2X5ZBCQ, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 188.166.225.170:55208->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297316256 (59h42m35.019879296s) [noise=true , address=104.36.82.10:4079 , piece_id=2JGXXC7DW3PFF3WZU7CU7S4MXAJ4ICMHSZJ4HKY33XC7EEPLLQEA, seg_pos={0 0}, upload=57278]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:25.297374718 (59h42m35.019937768s) [seg_pos={0 0}, upload=57278]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:18:25.531540137 (59h42m35.254103177s) [upload=57278]: Done uploading segments... Jul 26 23:18:25.531566388 (59h42m35.254129428s) [upload=57278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.236744456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X.X........X.X...........................XXXXXXXXXXXXXX] duration: 30.234423615s FAIL (2025-07-26 23:18:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.23444704s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.835386774s FAIL (2025-07-26 23:18:17 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.835411923s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XXX.X..X.....X] duration: 30.000760786s FAIL (2025-07-26 23:18:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000776927s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X.X........X.X...........................XXXXXXXXXXXXX] duration: 30.239286738s FAIL (2025-07-26 23:17:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.239307787s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.85852748s FAIL (2025-07-26 23:17:43 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.858546685s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235189012s FAIL (2025-07-26 23:17:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== iece_id=P2EB4FTEHYZNQOTEZUYJCKWPGGYI4OKTLLTYNLY6VOKSNBQDZMOA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.101246508 (59h40m48.823809548s) [noise=true , address=79.127.201.215:4360 , piece_id=HA5SDSTZT66XA3RYMDB7Y5NE44Y3P65LOWCFPW2O7D4BO5J26ZPA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.101605725 (59h40m48.824168765s) [noise=true , address=79.127.201.214:4321 , piece_id=OIL42A7QVMSCPEZI6D6YSPTTUEAFEHZUPZU7TELG5KYWYY3ADY3A, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.101652835 (59h40m48.824215875s) [noise=true , address=104.36.82.55:4194 , piece_id=CIYWLFM242PBGYBNJTFDSQJA4TVDENRNFZCTAYLBDNKM2MO2MR4Q, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.102153032 (59h40m48.824716072s) [noise=true , address=104.36.82.54:4191 , piece_id=LBQNYW34JFXHISZWAWOE55ISPDBVW3PJAXUT5VUNYEJPALCUNKTA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.102164485 (59h40m48.824727515s) [noise=true , address=79.127.205.235:4682 , piece_id=XEWSYPC4CUBC4GNDWXFHMNNOZGAC7K7NRJFFQV4LR7PK55XFDKMQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.102439030 (59h40m48.82500207s) [noise=true , address=79.127.219.38:4017 , piece_id=2L27YX7JSSW5U2ZKEOFMKLUR2SJWOYN6OUNFZIVQVJPPZGRRJH7Q, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.103321278 (59h40m48.825884318s) [noise=true , address=104.36.82.56:4253 , piece_id=CZEF6CF23HOROPA3AORZP46TKK4H7GTMEXSEX6ONHCNAPYNIFQGA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.103457328 (59h40m48.826020358s) [noise=true , address=79.127.205.245:4240 , piece_id=AMHSGVZC3NYB6SW6F3WQTCR4GLT76EFUNRPF53N7N4I3FILILT3Q, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.103958337 (59h40m48.826521367s) [noise=true , address=104.36.82.52:4091 , piece_id=RBWO4I6MEO2ZNHDFP6WY4GZ6BVP2CTGYNEC6RDJE33OOE2WKTYYQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.104324697 (59h40m48.826887737s) [noise=true , address=77.90.31.15:4723 , piece_id=SEFPWKWXVERRJ2TINBTDG7GMYK3YY7WGHNGZKW2WECRHNLY52CFQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.104343744 (59h40m48.826906784s) [noise=true , address=104.36.82.47:4530 , piece_id=ABBT2EMARL5LDMMBQKMGHKAPSFQD26T4FV5JEOHZF5RPJZV2KR5A, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.104601607 (59h40m48.827164647s) [noise=true , address=77.90.31.16:4743 , piece_id=4DWFOOWEY3QESIYUILZNTYVS6JY4AM3NZTJGJLXDIKB5L3EUEPHA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.104849772 (59h40m48.827412812s) [noise=true , address=104.36.82.46:4514 , piece_id=LUC4XUVBSPO66OT7M3VP3D3EBRIOOUGBI6MLJAFQ5ESAPAFD2QEQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.105089110 (59h40m48.82765215s) [noise=true , address=45.91.116.9:4273 , piece_id=NKAWJ5FDUUQWLRHZUNDTQF5O7F5GDGI6FVU2TRKWKATTIGE5246Q, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.105313409 (59h40m48.827876449s) [noise=true , address=104.36.82.41:4774 , piece_id=EO5GFI4ZPCYR5V3GZLUFNYCGRO2SYJXKE6PWHL4MPZS7J6FO2YTQ, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.105531607 (59h40m48.828094647s) [noise=true , address=45.91.116.11:4124 , piece_id=7LOHMHKC7JLNVDGXEKB5CYXR5ZOPT3Q5OZ2LT4RARSJPNTN267BA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:39.105759943 (59h40m48.828322983s) [noise=true , address=45.91.116.9:4270 , piece_id=B7YU2K7RSU2PLMK54AKIFJU5YA5ZHBXNDVX3PTTKKK5WK3ZLIAWA, seg_pos={0 0}, upload=57256]: Uploading piece... Jul 26 23:16:40.110039756 (59h40m49.832602796s) [noise=true , address=79.127.201.214:4321 , piece_id=OIL42A7QVMSCPEZI6D6YSPTTUEAFEHZUPZU7TELG5KYWYY3ADY3A, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:16:40.220391383 (59h40m49.942954423s) [upload=57256]: Next returned nil segment Jul 26 23:16:40.220409257 (59h40m49.942972297s) [upload=57256]: Waiting for error group managing segments... Jul 26 23:16:40.505659167 (59h40m50.228222207s) [noise=true , address=79.127.205.235:4682 , piece_id=XEWSYPC4CUBC4GNDWXFHMNNOZGAC7K7NRJFFQV4LR7PK55XFDKMQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:40.574667831 (59h40m50.297230871s) [noise=true , address=79.127.205.245:4240 , piece_id=AMHSGVZC3NYB6SW6F3WQTCR4GLT76EFUNRPF53N7N4I3FILILT3Q, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:40.967035320 (59h40m50.68959836s) [noise=true , address=79.127.219.38:4017 , piece_id=2L27YX7JSSW5U2ZKEOFMKLUR2SJWOYN6OUNFZIVQVJPPZGRRJH7Q, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:41.094575093 (59h40m50.817138133s) [noise=true , address=77.90.31.15:4723 , piece_id=SEFPWKWXVERRJ2TINBTDG7GMYK3YY7WGHNGZKW2WECRHNLY52CFQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:41.153069600 (59h40m50.87563263s) [noise=true , address=45.91.116.9:4270 , piece_id=B7YU2K7RSU2PLMK54AKIFJU5YA5ZHBXNDVX3PTTKKK5WK3ZLIAWA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:41.269502842 (59h40m50.992065882s) [noise=true , address=77.90.31.16:4743 , piece_id=4DWFOOWEY3QESIYUILZNTYVS6JY4AM3NZTJGJLXDIKB5L3EUEPHA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:41.279714380 (59h40m51.00227742s) [noise=true , address=45.91.116.9:4273 , piece_id=NKAWJ5FDUUQWLRHZUNDTQF5O7F5GDGI6FVU2TRKWKATTIGE5246Q, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:45.000563296 (59h40m54.723126336s) [noise=true , address=45.91.116.11:4124 , piece_id=7LOHMHKC7JLNVDGXEKB5CYXR5ZOPT3Q5OZ2LT4RARSJPNTN267BA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:16:45.732897706 (59h40m55.455460746s) [noise=true , address=79.127.220.101:4048 , piece_id=P2EB4FTEHYZNQOTEZUYJCKWPGGYI4OKTLLTYNLY6VOKSNBQDZMOA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:17:05.714163611 (59h41m15.436726661s) [noise=true , address=79.127.201.215:4360 , piece_id=HA5SDSTZT66XA3RYMDB7Y5NE44Y3P65LOWCFPW2O7D4BO5J26ZPA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: <nil> Jul 26 23:17:37.968191773 (59h41m47.690754813s) [noise=true , address=104.36.82.46:4514 , piece_id=LUC4XUVBSPO66OT7M3VP3D3EBRIOOUGBI6MLJAFQ5ESAPAFD2QEQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: rpc: dial tcp 104.36.82.46:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968261927 (59h41m47.690824967s) [noise=true , address=104.36.82.56:4253 , piece_id=CZEF6CF23HOROPA3AORZP46TKK4H7GTMEXSEX6ONHCNAPYNIFQGA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: write tcp 188.166.225.170:59058->104.36.82.56:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968307314 (59h41m47.690870354s) [noise=true , address=104.36.82.55:4194 , piece_id=CIYWLFM242PBGYBNJTFDSQJA4TVDENRNFZCTAYLBDNKM2MO2MR4Q, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: rpc: dial tcp 104.36.82.55:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968346399 (59h41m47.690909429s) [noise=true , address=104.36.82.47:4530 , piece_id=ABBT2EMARL5LDMMBQKMGHKAPSFQD26T4FV5JEOHZF5RPJZV2KR5A, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: write tcp 188.166.225.170:35900->104.36.82.47:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968369183 (59h41m47.690932223s) [noise=true , address=104.36.82.54:4191 , piece_id=LBQNYW34JFXHISZWAWOE55ISPDBVW3PJAXUT5VUNYEJPALCUNKTA, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: write tcp 188.166.225.170:42504->104.36.82.54:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968381686 (59h41m47.690944716s) [noise=true , address=104.36.82.52:4091 , piece_id=RBWO4I6MEO2ZNHDFP6WY4GZ6BVP2CTGYNEC6RDJE33OOE2WKTYYQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: write tcp 188.166.225.170:33386->104.36.82.52:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968392928 (59h41m47.690955968s) [noise=true , address=104.36.82.41:4774 , piece_id=EO5GFI4ZPCYR5V3GZLUFNYCGRO2SYJXKE6PWHL4MPZS7J6FO2YTQ, seg_pos={0 0}, upload=57256]: Done uploading piece. err: ecclient: upload failed (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb, address:104.36.82.41:4774): protocol: rpc: dial tcp 104.36.82.41:4774: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:37.968409629 (59h41m47.69097267s) [seg_pos={0 0}, upload=57256]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:17:38.201427694 (59h41m47.923990734s) [upload=57256]: Done uploading segments... Jul 26 23:17:38.201453774 (59h41m47.924016804s) [upload=57256]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.235200333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.............X..XX...X.X.X.X.......XX..XX.XX..XXX...........] duration: 10.119631004s FAIL (2025-07-26 23:17:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.119641414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXX.X..X.....] duration: 26.164022328s FAIL (2025-07-26 23:17:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.164040662s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [..............X..XX...X.X.X.X.......XX..XX.XX..XXX..........] duration: 30.898734644s FAIL (2025-07-26 23:17:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.898751226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...............X.........X...........X.......XXXXXXXXXXXXXXX] duration: 30.240002439s FAIL (2025-07-26 23:17:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 47.996956281 (59h40m57.719519311s) [upload=57250]: Done uploading segments... Jul 26 23:16:47.996981249 (59h40m57.719544289s) [upload=57250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:16:48.872851333 (59h40m58.595414373s) [upload=57260]: Starting upload Jul 26 23:16:48.872927048 (59h40m58.595490088s) [upload=57260]: Done starting upload Jul 26 23:16:48.873029574 (59h40m58.595592614s) [upload=57260]: Uploading segments... Jul 26 23:16:48.873142911 (59h40m58.595705951s) [upload=57260]: Got next segment. Inline: false Jul 26 23:16:48.873180984 (59h40m58.595744024s) [upload=57260]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:16:48.873187096 (59h40m58.595750126s) [upload=57260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:16:49.343403844 (59h40m59.065966884s) [seg_pos={0 0}, upload=57260]: Begin upload segment... Jul 26 23:16:49.343447838 (59h40m59.066010878s) [seg_pos={0 0}, upload=57260]: Done begin upload segment. Jul 26 23:16:49.343470662 (59h40m59.066033692s) [noise=true , address=45.91.116.7:4011 , piece_id=MXU2WJATFMMHQUHMFCHPHPMI7QR2R5D67PHLO2M5KFN3V6VHC72A, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.343514806 (59h40m59.066077846s) [noise=true , address=77.90.31.14:4661 , piece_id=4ZOGBM2UN4LVQMI26S3VSKNFQNRJE5UGL3AZEIRFPZZ7QL63QYIA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.343834508 (59h40m59.066397538s) [noise=true , address=45.91.116.8:4045 , piece_id=KJ3EZGGLW3DDDXUVWXAQIF5NIWUC2EVAPUG33TAW4D2YN75OKPOA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.344250774 (59h40m59.066813814s) [noise=true , address=104.36.82.10:4078 , piece_id=4L4Y4KPKVXESEYSZP6Q26R5J7V2CSXUNEI2SE7GB7B3A3OJBC4VQ, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.344703571 (59h40m59.067266611s) [noise=true , address=45.91.116.12:4226 , piece_id=LYRTQQKREHKRKBVPFOR3AVIXQB6SVJT4ZECKKOH23WNRMZTGO4OA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.344823220 (59h40m59.0673863s) [noise=true , address=104.36.82.10:4079 , piece_id=7IMEBZTMVUAAX2MY6HU3FSZJSDT72RHJY4FYP2RWVWHFRBDVJZAA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.344968708 (59h40m59.067531748s) [noise=true , address=45.91.116.10:4083 , piece_id=EK2S656KNAC7MSJGYK3FAL4ZJOBP4FF26VRPXHKPJGWJLM7JQFAQ, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345214779 (59h40m59.067777819s) [noise=true , address=77.90.31.10:4518 , piece_id=SJSEZPJQNAV4NXEMIOCNKZTW6RR7HXHWZ7C3BWJAEOJA3OSFPHTQ, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345458385 (59h40m59.068021425s) [noise=true , address=104.36.82.6:4876 , piece_id=IQB4KI5VC3UXWREOJLYV3FOEEPEZ42YTQI7W73Z37ZTQYCG2MK4A, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345689457 (59h40m59.068252497s) [noise=true , address=45.91.116.7:4009 , piece_id=AWFZDBRE4HGD3BOPUU7JITDWE4Y67W3CV7U4SH5BDFNQILUZA6SA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345922343 (59h40m59.068485383s) [noise=true , address=45.91.116.5:4298 , piece_id=X6DEY45NFWRHQPHUODFEB7WFWRKE6VIACDRJE2D2L4VYXHIWQEVA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.345946228 (59h40m59.068509269s) [noise=true , address=104.36.82.15:4258 , piece_id=776ZFSJQMWJI6WVFJ6O24X42S4TBIXQMVERNBSKDAYSBDIAAITCA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.346186427 (59h40m59.068749467s) [noise=true , address=104.36.82.12:4149 , piece_id=SQYE3VKF4ISMBZCOJXPEWFITEQWYQ4VYZOFUDYLH7KIOL34DJEAA, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.346422309 (59h40m59.068985339s) [noise=true , address=45.91.116.4:4938 , piece_id=2NQNCSG2HOYBPHVDMPE4SXXGZJX2NO4UAOSLM6QWPXYSFO5NB43A, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.346652569 (59h40m59.069215609s) [noise=true , address=104.36.82.7:4909 , piece_id=QYXGSAB53RBNZ7GSWEBYLHSB2ZTYS4UVJHZ5LFHRFF2F7DYV4NSQ, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:49.346876579 (59h40m59.069439619s) [noise=true , address=45.91.116.6:4973 , piece_id=7BXFIPGIZXJW6WJ4XZ5BK5LDURPHVEKG2ELT7H6XXAIDPLKMLX3A, seg_pos={0 0}, upload=57260]: Uploading piece... Jul 26 23:16:51.044906306 (59h41m0.767469346s) [upload=57260]: Next returned nil segment Jul 26 23:16:51.044926395 (59h41m0.767489435s) [upload=57260]: Waiting for error group managing segments... Jul 26 23:16:51.364934428 (59h41m1.087497458s) [noise=true , address=45.91.116.5:4298 , piece_id=X6DEY45NFWRHQPHUODFEB7WFWRKE6VIACDRJE2D2L4VYXHIWQEVA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.372466910 (59h41m1.09502995s) [noise=true , address=45.91.116.7:4009 , piece_id=AWFZDBRE4HGD3BOPUU7JITDWE4Y67W3CV7U4SH5BDFNQILUZA6SA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.459560226 (59h41m1.182123266s) [noise=true , address=45.91.116.8:4045 , piece_id=KJ3EZGGLW3DDDXUVWXAQIF5NIWUC2EVAPUG33TAW4D2YN75OKPOA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.463646744 (59h41m1.186209784s) [noise=true , address=45.91.116.6:4973 , piece_id=7BXFIPGIZXJW6WJ4XZ5BK5LDURPHVEKG2ELT7H6XXAIDPLKMLX3A, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.514342059 (59h41m1.23690509s) [noise=true , address=45.91.116.4:4938 , piece_id=2NQNCSG2HOYBPHVDMPE4SXXGZJX2NO4UAOSLM6QWPXYSFO5NB43A, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.522099993 (59h41m1.244663033s) [noise=true , address=45.91.116.12:4226 , piece_id=LYRTQQKREHKRKBVPFOR3AVIXQB6SVJT4ZECKKOH23WNRMZTGO4OA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.534967702 (59h41m1.257530732s) [noise=true , address=77.90.31.14:4661 , piece_id=4ZOGBM2UN4LVQMI26S3VSKNFQNRJE5UGL3AZEIRFPZZ7QL63QYIA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.551777084 (59h41m1.274340125s) [noise=true , address=45.91.116.10:4083 , piece_id=EK2S656KNAC7MSJGYK3FAL4ZJOBP4FF26VRPXHKPJGWJLM7JQFAQ, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:16:51.612576742 (59h41m1.335139772s) [noise=true , address=45.91.116.7:4011 , piece_id=MXU2WJATFMMHQUHMFCHPHPMI7QR2R5D67PHLO2M5KFN3V6VHC72A, seg_pos={0 0}, upload=57260]: Done uploading piece. err: <nil> Jul 26 23:17:18.232454490 (59h41m27.95501753s) [noise=true , address=77.90.31.10:4518 , piece_id=SJSEZPJQNAV4NXEMIOCNKZTW6RR7HXHWZ7C3BWJAEOJA3OSFPHTQ, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232501430 (59h41m27.95506446s) [noise=true , address=104.36.82.7:4909 , piece_id=QYXGSAB53RBNZ7GSWEBYLHSB2ZTYS4UVJHZ5LFHRFF2F7DYV4NSQ, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:40490->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232613455 (59h41m27.955176495s) [noise=true , address=104.36.82.12:4149 , piece_id=SQYE3VKF4ISMBZCOJXPEWFITEQWYQ4VYZOFUDYLH7KIOL34DJEAA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232642941 (59h41m27.955205981s) [noise=true , address=104.36.82.6:4876 , piece_id=IQB4KI5VC3UXWREOJLYV3FOEEPEZ42YTQI7W73Z37ZTQYCG2MK4A, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:57838->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232665414 (59h41m27.955228454s) [noise=true , address=104.36.82.10:4078 , piece_id=4L4Y4KPKVXESEYSZP6Q26R5J7V2CSXUNEI2SE7GB7B3A3OJBC4VQ, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232675724 (59h41m27.955238764s) [noise=true , address=104.36.82.15:4258 , piece_id=776ZFSJQMWJI6WVFJ6O24X42S4TBIXQMVERNBSKDAYSBDIAAITCA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232679040 (59h41m27.95524207s) [noise=true , address=104.36.82.10:4079 , piece_id=7IMEBZTMVUAAX2MY6HU3FSZJSDT72RHJY4FYP2RWVWHFRBDVJZAA, seg_pos={0 0}, upload=57260]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:18.232698868 (59h41m27.955261908s) [seg_pos={0 0}, upload=57260]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:17:18.470668807 (59h41m28.193231837s) [upload=57260]: Done uploading segments... Jul 26 23:17:18.470692984 (59h41m28.193256024s) [upload=57260]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.240014192s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)