Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [..............................X...XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23433276s FAIL (2025-07-28 00:35:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== , piece_id=RPHSPO2JQMKG6EEL2G4FSCMEESCMQGHJWNMQH6PW5XYELR6PABMQ, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.186646093 (84h58m20.909209133s) [noise=true , address=64.49.13.154:4932 , piece_id=ILL43ANVYOBRISC3QMBR6JTCF67DMQG6HC2CNJFJMIPCBTB7KLAQ, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.187015289 (84h58m20.909578339s) [noise=true , address=104.36.82.6:4894 , piece_id=AUZEUHAM2XVKRRK5AGS35JKSHVXCPYBYPVB53VZ2UFKPPRHMQHIA, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.187865666 (84h58m20.910428706s) [noise=true , address=104.36.82.15:4263 , piece_id=XWWC3B4EX5CE7Q2S5QCJAQJRGQNT4KICA56QMMYJHHPNSQVDQZ7Q, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.188578550 (84h58m20.91114159s) [noise=true , address=104.36.82.14:4242 , piece_id=6XHODN4FXR4L7KDIVRI7XZ43JSQBD6E26HWR5DXS6XBTNWPNKUFA, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.193602393 (84h58m20.916165433s) [noise=true , address=104.36.82.3:4783 , piece_id=C3E35WFHLV6QATOFQNFEEH3CCITV35TOUOKTQ5BWSVQLKWXKHQDA, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.194456497 (84h58m20.917019537s) [noise=true , address=104.36.82.51:4059 , piece_id=WAAFHIH5XM6WT7N3WLZ3YLTOMPILZADKLJQYZEWLHX2ENRJ5VCMA, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:12.331374429 (84h58m22.053937469s) [upload=81541]: Next returned nil segment Jul 28 00:34:12.331391842 (84h58m22.053954883s) [upload=81541]: Waiting for error group managing segments... Jul 28 00:34:12.640722704 (84h58m22.363285744s) [noise=true , address=64.49.13.154:4932 , piece_id=ILL43ANVYOBRISC3QMBR6JTCF67DMQG6HC2CNJFJMIPCBTB7KLAQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: <nil> Jul 28 00:34:13.747173143 (84h58m23.469736183s) [noise=true , address=77.90.31.16:4748 , piece_id=5PMV6PPODQQENZMCG7OXUKI7Q36XJI7JDX6D5B2ULEQWG5XJJE6A, seg_pos={0 0}, upload=81541]: Done uploading piece. err: <nil> Jul 28 00:35:10.081128191 (84h59m19.803691241s) [noise=true , address=104.36.82.35:4615 , piece_id=PDBNLHVRWOGD6VQRYUXRARQFJKXBUANFL6YAUAJHXLFRPBNO23LA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: write tcp 188.166.225.170:50616->104.36.82.35:4615: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081179818 (84h59m19.803742859s) [noise=true , address=104.36.82.13:4193 , piece_id=RPHSPO2JQMKG6EEL2G4FSCMEESCMQGHJWNMQH6PW5XYELR6PABMQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:15XNLgptmXprcfkVMJsuzY2WDyHiZDpLetiBpmjXhb4c9q7yMd, address:104.36.82.13:4193): protocol: write tcp 188.166.225.170:60864->104.36.82.13:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081214205 (84h59m19.803777235s) [noise=true , address=104.36.82.6:4894 , piece_id=AUZEUHAM2XVKRRK5AGS35JKSHVXCPYBYPVB53VZ2UFKPPRHMQHIA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12fZsj77XZ8CnYD6iwzEVKPn6yLyxWMtibzsRXcuGKwNxEXB6Hq, address:104.36.82.6:4894): protocol: write tcp 188.166.225.170:38976->104.36.82.6:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081249612 (84h59m19.803812653s) [noise=true , address=104.36.82.53:4154 , piece_id=LMMA7JHS2NGKKX3OB3MNWYCM3UPNGPQLXS347PCXYDLLPBNZWOOQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: write tcp 188.166.225.170:52900->104.36.82.53:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081281223 (84h59m19.803844253s) [noise=true , address=104.36.82.12:4154 , piece_id=GQYNCE7AITKLZ7DOHQNZ7LYKEXAJVI2FMUT6IBXCJPIL3UGFM36A, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: write tcp 188.166.225.170:47294->104.36.82.12:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081313064 (84h59m19.803876094s) [noise=true , address=104.36.82.55:4193 , piece_id=HYOCXSRM47ZMO7QUU35JTPU3GVLGSF6TWWEB3ZEUUD3YHFFO5VVA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: rpc: dial tcp 104.36.82.55:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081347079 (84h59m19.803910109s) [noise=true , address=104.36.82.14:4242 , piece_id=6XHODN4FXR4L7KDIVRI7XZ43JSQBD6E26HWR5DXS6XBTNWPNKUFA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: write tcp 188.166.225.170:59762->104.36.82.14:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081382457 (84h59m19.803945487s) [noise=true , address=104.36.82.51:4059 , piece_id=WAAFHIH5XM6WT7N3WLZ3YLTOMPILZADKLJQYZEWLHX2ENRJ5VCMA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: write tcp 188.166.225.170:33624->104.36.82.51:4059: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081416692 (84h59m19.803979732s) [noise=true , address=104.36.82.54:4191 , piece_id=FJXF54KAMXXFSJPM5DGH56RTPCSLTGMFEBBJRTK3FLK4UIF4VKDQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: rpc: dial tcp 104.36.82.54:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081457481 (84h59m19.804020521s) [noise=true , address=104.36.82.3:4783 , piece_id=C3E35WFHLV6QATOFQNFEEH3CCITV35TOUOKTQ5BWSVQLKWXKHQDA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12ito72D7vikRdjK7BPt4LnhAK84bzi8AbZZPNWrTqmbjwQT1N, address:104.36.82.3:4783): protocol: write tcp 188.166.225.170:42850->104.36.82.3:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081493920 (84h59m19.80405696s) [noise=true , address=104.36.82.52:4090 , piece_id=73SMTCFCCI76TX3TG2SYNSDPTBS5NOCKEG5HAVKKB7DKNSY464ZA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 188.166.225.170:42426->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081533115 (84h59m19.804096155s) [noise=true , address=104.36.82.12:4169 , piece_id=443NIAZIZBAABGE64P7H5RC3J4VN3EBYEVOLDV6UIZFDR3NOSJIA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: write tcp 188.166.225.170:35794->104.36.82.12:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081550899 (84h59m19.804113939s) [noise=true , address=104.36.82.13:4220 , piece_id=WPBF76CZLFTULBAUXCRUKY7NUB65GXFWIDD3YF6ADASPSD723E6A, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1AjQmw3H58hTwr2389cmMcwy1gBavBrZB8AWB1gqfoFRUkCpH6, address:104.36.82.13:4220): protocol: write tcp 188.166.225.170:33048->104.36.82.13:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081578412 (84h59m19.804141442s) [noise=true , address=104.36.82.42:4818 , piece_id=T4NRYW7NX2KKQTCXRWAGUJOY354GLZEMKHHYEABPSJZ2TXNP47IQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:104.36.82.42:4818): protocol: write tcp 188.166.225.170:55632->104.36.82.42:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081616073 (84h59m19.804179113s) [noise=true , address=104.36.82.56:4233 , piece_id=3VTZ3QJO5AYNKPGJX34ZN6QGJFXGMDO4J7MPJ4K3ISOK3PS3ARHQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: write tcp 188.166.225.170:46066->104.36.82.56:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081644908 (84h59m19.804207948s) [noise=true , address=104.36.82.15:4263 , piece_id=XWWC3B4EX5CE7Q2S5QCJAQJRGQNT4KICA56QMMYJHHPNSQVDQZ7Q, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:45518->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081692550 (84h59m19.80425559s) [seg_pos={0 0}, upload=81541]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:35:10.313368144 (84h59m20.035931184s) [upload=81541]: Done uploading segments... Jul 28 00:35:10.313395426 (84h59m20.035958466s) [upload=81541]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.234345685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............X............X.......................X........] duration: 1.962413981s FAIL (2025-07-28 00:35:04 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.962447054s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X.X..........X.....X.X........] duration: 5.841326255s FAIL (2025-07-28 00:34:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841350662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................X.......X.XX....X....X.......] duration: 5.717905685s FAIL (2025-07-28 00:34:32 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 5.717930061s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X.X..........X.....X.X.......] duration: 5.846990585s FAIL (2025-07-28 00:34:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.847018258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............X............X.......................X.......] duration: 1.887216839s FAIL (2025-07-28 00:34:30 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.88724369s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [...............................X...XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236287612s FAIL (2025-07-28 00:34:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 9s) [noise=true , address=104.36.82.41:4792 , piece_id=SYECFTFRXUXTV3AH6S5ZCCJDJ4FK55QSM3AZGBJI2KYJQJOCRG6A, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.698765679 (84h57m20.421328719s) [noise=true , address=104.36.82.54:4178 , piece_id=6K67LI7ORQPN55ZAFMYCLERDPELOBWD23J7MLY75BV5RI5OECLVQ, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.698981862 (84h57m20.421544892s) [noise=true , address=104.36.82.10:4094 , piece_id=RRAIL44JAI3S7YKHWDKWSAXASHUKUUIE3LCTQNJ4VBAFEPR2UMNQ, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.699198977 (84h57m20.421762017s) [noise=true , address=79.127.205.235:4680 , piece_id=IF37CXAAWWSYVS35WBU673TAOYAYOHCYMXSWKP54V2POL6PLLGRQ, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.699336190 (84h57m20.42189923s) [noise=true , address=104.36.82.12:4173 , piece_id=QA3LG5NAHCK6WN5CB3RNJHTUU2LFALXIIGLLV3NPN4NOYDRBGJAQ, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.699590717 (84h57m20.422153767s) [noise=true , address=104.36.82.49:4996 , piece_id=OXYDIPLEPI75BALZQP4WXT4Z624EZGHN5PXGGRAHQDWZRP7HHQPQ, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.699803113 (84h57m20.422366153s) [noise=true , address=104.36.82.56:4263 , piece_id=HCBLSZYV2FB3G2DMO6J2CAAIXASU753PCB5CDWT5O4J7SF2JNJKQ, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.700089311 (84h57m20.422652351s) [noise=true , address=104.36.82.12:4164 , piece_id=GR5PC75RIVY2GFKLH2BYRT6LJUZQ2A3UJJWZ5H7VI5MD62PYJDKQ, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.700251562 (84h57m20.422814602s) [noise=true , address=104.36.82.5:4865 , piece_id=KPDVEP3BOJBSESMSVFXLDRAQCTBJIQIJVAXQF7J67G2KJYP46MPA, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:10.700531728 (84h57m20.423094768s) [noise=true , address=104.36.82.11:4130 , piece_id=QDL2JDDN66JIH3WJK2SSH4TNGLEAULE4QCWAO3FUQP5IW3B7SGPA, seg_pos={0 0}, upload=81524]: Uploading piece... Jul 28 00:33:11.826597687 (84h57m21.549160727s) [upload=81524]: Next returned nil segment Jul 28 00:33:11.826616423 (84h57m21.549179463s) [upload=81524]: Waiting for error group managing segments... Jul 28 00:33:12.248980612 (84h57m21.971543652s) [noise=true , address=79.127.205.235:4680 , piece_id=IF37CXAAWWSYVS35WBU673TAOYAYOHCYMXSWKP54V2POL6PLLGRQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: <nil> Jul 28 00:33:16.395924860 (84h57m26.1184879s) [noise=true , address=64.49.13.140:4508 , piece_id=LL5WBCBIBQBYD2YJU6YLGM7KNMDAFPZWCJZ5RLHZ6JS46762VDYQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: <nil> Jul 28 00:33:30.687648510 (84h57m40.41021155s) [noise=true , address=104.36.82.5:4865 , piece_id=KPDVEP3BOJBSESMSVFXLDRAQCTBJIQIJVAXQF7J67G2KJYP46MPA, seg_pos={0 0}, upload=81524]: Done uploading piece. err: <nil> Jul 28 00:34:09.609715024 (84h58m19.332278084s) [noise=true , address=104.36.82.56:4263 , piece_id=HCBLSZYV2FB3G2DMO6J2CAAIXASU753PCB5CDWT5O4J7SF2JNJKQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: rpc: dial tcp 104.36.82.56:4263: i/o timeout; context deadline exceeded Jul 28 00:34:09.610471942 (84h58m19.333035022s) [noise=true , address=104.36.82.11:4130 , piece_id=QDL2JDDN66JIH3WJK2SSH4TNGLEAULE4QCWAO3FUQP5IW3B7SGPA, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:12n69riZBquh3Qi436LVfVgNDJYuDxPzGi7d2BZDLurUJYVMCqs, address:104.36.82.11:4130): protocol: rpc: dial tcp 104.36.82.11:4130: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.610611318 (84h58m19.333174368s) [noise=true , address=104.36.82.52:4106 , piece_id=CEHG3GGYKBL6ZSHOJZKPGOUSPVGSR5765L7FZIMOC4TOSXCC2OZQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.610671483 (84h58m19.333234563s) [noise=true , address=104.36.82.14:4247 , piece_id=3XEEW3WS2PRADEGMF7ZKOIBRK7FD2UN6WWUYYEBQ5FEKUCQEUBNA, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:104.36.82.14:4247): protocol: rpc: dial tcp 104.36.82.14:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.610722090 (84h58m19.33328514s) [noise=true , address=104.36.82.41:4792 , piece_id=SYECFTFRXUXTV3AH6S5ZCCJDJ4FK55QSM3AZGBJI2KYJQJOCRG6A, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: rpc: dial tcp 104.36.82.41:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.610781313 (84h58m19.333344353s) [noise=true , address=104.36.82.10:4094 , piece_id=RRAIL44JAI3S7YKHWDKWSAXASHUKUUIE3LCTQNJ4VBAFEPR2UMNQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:124zBCfZBTW69BSaGRzUyjWUBjSJhaLMZXQYnqMPGSKL4EX6HVV, address:104.36.82.10:4094): protocol: rpc: dial tcp 104.36.82.10:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.610840718 (84h58m19.333403758s) [noise=true , address=104.36.82.16:4326 , piece_id=GBPPEWEQD4D7XDYHUDHPEFQJWTAWE4MAXMTKHARSUQCBCDFMBMCQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:12pU8MZZKS88CPFdZoaQ4msNfkCbCewTmS2wKEdN7pG5aCWoAr6, address:104.36.82.16:4326): protocol: write tcp 188.166.225.170:53392->104.36.82.16:4326: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.610884351 (84h58m19.333447391s) [noise=true , address=104.36.82.20:4441 , piece_id=KXJP6GVDT7FA5BA4ZNK4D24I7BOJY34SMYP6WRECL5JWKCMOFLJQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:104.36.82.20:4441): protocol: rpc: dial tcp 104.36.82.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.611012045 (84h58m19.333575085s) [noise=true , address=104.36.82.53:4135 , piece_id=GPY2PZVGUSYDVBBXIQB2PKLKW63OT3BHC5QJ72Y6OSRMBRGJYK6Q, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 188.166.225.170:50118->104.36.82.53:4135: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.611055538 (84h58m19.333618588s) [noise=true , address=104.36.82.12:4173 , piece_id=QA3LG5NAHCK6WN5CB3RNJHTUU2LFALXIIGLLV3NPN4NOYDRBGJAQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:12HbGUhwH2eNqY75sLQCDgkvJ1QSrJ2n4Qg8A5Z73dFXxRpVwSC, address:104.36.82.12:4173): protocol: rpc: dial tcp 104.36.82.12:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.611085596 (84h58m19.333648647s) [noise=true , address=104.36.82.12:4164 , piece_id=GR5PC75RIVY2GFKLH2BYRT6LJUZQ2A3UJJWZ5H7VI5MD62PYJDKQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: rpc: dial tcp 104.36.82.12:4164: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.611097218 (84h58m19.333660268s) [noise=true , address=104.36.82.49:4996 , piece_id=OXYDIPLEPI75BALZQP4WXT4Z624EZGHN5PXGGRAHQDWZRP7HHQPQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:1qomJ2ceswE6SKfoPAxWpiCkcW88EpfFbPnJXs7ETb5CZvSpqS, address:104.36.82.49:4996): protocol: rpc: dial tcp 104.36.82.49:4996: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.611133578 (84h58m19.333696618s) [noise=true , address=104.36.82.7:4944 , piece_id=DQCQME2PIXJ4UDP4N4V6UBE2VEJDV7ZWSCW2QC45JMDM2UZIJJQQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: rpc: dial tcp 104.36.82.7:4944: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.611141002 (84h58m19.333704043s) [noise=true , address=104.36.82.54:4178 , piece_id=6K67LI7ORQPN55ZAFMYCLERDPELOBWD23J7MLY75BV5RI5OECLVQ, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:104.36.82.54:4178): protocol: write tcp 188.166.225.170:40244->104.36.82.54:4178: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.611170579 (84h58m19.333733629s) [noise=true , address=104.36.82.55:4193 , piece_id=MV2WZRSVVX5AWC26YJ5IWSZLZZC6J7DYZLOMYYDBG7SX572HLKCA, seg_pos={0 0}, upload=81524]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: write tcp 188.166.225.170:46240->104.36.82.55:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:34:09.611186980 (84h58m19.33375003s) [seg_pos={0 0}, upload=81524]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:34:09.844596272 (84h58m19.567159322s) [upload=81524]: Done uploading segments... Jul 28 00:34:09.844633002 (84h58m19.567196042s) [upload=81524]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.236300356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X............X.......................X......] duration: 1.823046592s FAIL (2025-07-28 00:34:01 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.823074476s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................X.......X.XX....X....X......] duration: 5.809342664s FAIL (2025-07-28 00:33:54 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 5.809368495s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X.X..........X.....X.X......] duration: 6.025388043s FAIL (2025-07-28 00:33:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.025417068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................X.......X.XX....X....X.....] duration: 5.939563164s FAIL (2025-07-28 00:33:11 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 5.939586377s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [................................X...XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232325616s FAIL (2025-07-28 00:33:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ]: Uploading piece... Jul 28 00:32:10.247040697 (84h56m19.969603727s) [noise=true , address=104.36.82.11:4130 , piece_id=VYUO44ZM5IOUADNGVGOORTCDTYTDC7LVZNBTHYWLJCCVAR3TPUNQ, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:10.247281317 (84h56m19.969844357s) [noise=true , address=104.36.82.50:4030 , piece_id=JZWXUSRMZ6HHEQADD5J24QD2KNS4RAE6UFZQ26RRK6VD6DCEF6BQ, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:10.247305313 (84h56m19.969868353s) [noise=true , address=104.36.82.42:4817 , piece_id=ZLQ2EQ3356DTP2BTE7OFN25EXPW3VKPDM6R4SUVQLTWG2IKP3VFQ, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:10.247511738 (84h56m19.970074778s) [noise=true , address=104.36.82.15:4263 , piece_id=ZCVEL5HGPTXWWMPJBNLBFNZWE3ZBHU6PJ3XUS2UNLZCYEY2HIAAA, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:10.247536836 (84h56m19.970099876s) [noise=true , address=104.36.82.36:4632 , piece_id=4O3MSSD6D2IUZMFTJKUNDI6W3MZ7G6UQEZD7YPY4GHXBLA6SSNSA, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:10.247825278 (84h56m19.970388318s) [noise=true , address=64.49.13.138:4482 , piece_id=GSLBSUVOYRBBDT2WKXTCFNGKU6LOOOBYHYF4PJOLK4JFCQHSTX3A, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:10.248058524 (84h56m19.970621564s) [noise=true , address=104.36.82.14:4245 , piece_id=SFMTOCGS5RN56AOIWSZQA5KCBYYIZAMSYPFB6AFLTQETF4GVUXOA, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:10.248285448 (84h56m19.970848488s) [noise=true , address=104.36.82.47:4517 , piece_id=7HMOHZQLHTVAVR3DZNQ6L45A7732BAXA5XPAXLK2HKM7XP473TUQ, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:10.248562929 (84h56m19.971125969s) [noise=true , address=79.127.201.215:4347 , piece_id=N5PAJS3TL6VCVQM5HGQU7OVOT4GSMKOGAYMB6Q3O4FVWLTRMFXCQ, seg_pos={0 0}, upload=81507]: Uploading piece... Jul 28 00:32:11.392616244 (84h56m21.115179284s) [upload=81507]: Next returned nil segment Jul 28 00:32:11.392633728 (84h56m21.115196768s) [upload=81507]: Waiting for error group managing segments... Jul 28 00:32:12.055979967 (84h56m21.778543007s) [noise=true , address=64.49.13.138:4482 , piece_id=GSLBSUVOYRBBDT2WKXTCFNGKU6LOOOBYHYF4PJOLK4JFCQHSTX3A, seg_pos={0 0}, upload=81507]: Done uploading piece. err: <nil> Jul 28 00:32:12.253720744 (84h56m21.976283784s) [noise=true , address=79.127.201.215:4347 , piece_id=N5PAJS3TL6VCVQM5HGQU7OVOT4GSMKOGAYMB6Q3O4FVWLTRMFXCQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: <nil> Jul 28 00:33:09.147055765 (84h57m18.869618795s) [noise=true , address=104.36.82.50:4030 , piece_id=JZWXUSRMZ6HHEQADD5J24QD2KNS4RAE6UFZQ26RRK6VD6DCEF6BQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1fRTqHVZiYV7NBAp7PLEPVXGEDZ7VXxQJuRhkXkAkNpT7ra27e, address:104.36.82.50:4030): protocol: rpc: dial tcp 104.36.82.50:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147127042 (84h57m18.869690082s) [noise=true , address=104.36.82.9:4009 , piece_id=C5AKHDCSQTO2MFAZKFWU6ONY7QARLLSDBVWI6664FUZ7HMLN6XRQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147160305 (84h57m18.869723345s) [noise=true , address=104.36.82.55:4193 , piece_id=3SE7KZRBBMRPDHKE3AXRXINAF3HA5GHXXOJQC5GC3FPPQ77SXZQQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: rpc: dial tcp 104.36.82.55:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147188800 (84h57m18.86975183s) [noise=true , address=104.36.82.47:4517 , piece_id=7HMOHZQLHTVAVR3DZNQ6L45A7732BAXA5XPAXLK2HKM7XP473TUQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147219478 (84h57m18.869782518s) [noise=true , address=104.36.82.43:4868 , piece_id=RKAR33A2AALB3KK6OFGC3EJ6I7B6NCSA3AAUXS3FHADSS6ZLXFVA, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:124LJtYshPwu1uZJwVyjrjUMk3htyVnp9ntnZWSUb7X3bNkwU6t, address:104.36.82.43:4868): protocol: rpc: dial tcp 104.36.82.43:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147248574 (84h57m18.869811614s) [noise=true , address=104.36.82.45:4936 , piece_id=2NH3CNWI5GZROOG5LQ5K4R5OUFWXB4CC6RDW3OQBL3K4NSRQVHTA, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:104.36.82.45:4936): protocol: rpc: dial tcp 104.36.82.45:4936: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147275666 (84h57m18.869838706s) [noise=true , address=104.36.82.53:4147 , piece_id=HF6QO6SDK2ZREDIUY5R76PVPG3DLFY3JWFAGLIKVJMN4HUNMFNGQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147301094 (84h57m18.869864134s) [noise=true , address=104.36.82.20:4454 , piece_id=5AKBK5Z2HCDV727LZZOT2AMWO2XFXSNSYKXU6EMENBZFLLJWTAIA, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:104.36.82.20:4454): protocol: rpc: dial tcp 104.36.82.20:4454: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147328767 (84h57m18.869891807s) [noise=true , address=104.36.82.6:4885 , piece_id=MRXBZKBIG34JYNKUFNYPH2YRRQ77DHZ2OKQUN7RCOYEDR6GCRM3Q, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:12rLd1AWyRtEVFmK4iJCebp5VAJEjnmhhgtUQcV1FB8vMvRnUhm, address:104.36.82.6:4885): protocol: rpc: dial tcp 104.36.82.6:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147360017 (84h57m18.869923057s) [noise=true , address=104.36.82.42:4817 , piece_id=ZLQ2EQ3356DTP2BTE7OFN25EXPW3VKPDM6R4SUVQLTWG2IKP3VFQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: rpc: dial tcp 104.36.82.42:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147388000 (84h57m18.869951041s) [noise=true , address=104.36.82.15:4263 , piece_id=ZCVEL5HGPTXWWMPJBNLBFNZWE3ZBHU6PJ3XUS2UNLZCYEY2HIAAA, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: rpc: dial tcp 104.36.82.15:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147415945 (84h57m18.869978985s) [noise=true , address=104.36.82.36:4632 , piece_id=4O3MSSD6D2IUZMFTJKUNDI6W3MZ7G6UQEZD7YPY4GHXBLA6SSNSA, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: rpc: dial tcp 104.36.82.36:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147443468 (84h57m18.870006508s) [noise=true , address=104.36.82.54:4165 , piece_id=XCPWPVRFMF24SLMII6GKHLDKSY3MOLJWHLTZFYS6XXDUUV5AXIYQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147468785 (84h57m18.870031825s) [noise=true , address=104.36.82.41:4762 , piece_id=HAFZBR3IO3TS6APFB36QQMHA4GNXANW36R662MQR4STZWDMABHAQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:104.36.82.41:4762): protocol: rpc: dial tcp 104.36.82.41:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147494705 (84h57m18.870057745s) [noise=true , address=104.36.82.11:4130 , piece_id=VYUO44ZM5IOUADNGVGOORTCDTYTDC7LVZNBTHYWLJCCVAR3TPUNQ, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:12n69riZBquh3Qi436LVfVgNDJYuDxPzGi7d2BZDLurUJYVMCqs, address:104.36.82.11:4130): protocol: rpc: dial tcp 104.36.82.11:4130: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147526757 (84h57m18.870089797s) [noise=true , address=104.36.82.14:4245 , piece_id=SFMTOCGS5RN56AOIWSZQA5KCBYYIZAMSYPFB6AFLTQETF4GVUXOA, seg_pos={0 0}, upload=81507]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: write tcp 188.166.225.170:48656->104.36.82.14:4245: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:33:09.147540332 (84h57m18.870103372s) [seg_pos={0 0}, upload=81507]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:33:09.376631782 (84h57m19.099194882s) [upload=81507]: Done uploading segments... Jul 28 00:33:09.376685816 (84h57m19.099248856s) [upload=81507]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.232349221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X.X..........X.....X.X.....] duration: 5.76470518s FAIL (2025-07-28 00:32:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.76473707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X............X.......................X.....] duration: 1.896496482s FAIL (2025-07-28 00:32:46 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.896522303s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X............X.......................X....] duration: 1.854090328s FAIL (2025-07-28 00:32:26 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.854123491s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................X.......X.XX....X....X....] duration: 6.029700435s FAIL (2025-07-28 00:32:25 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 6.029727807s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................X............X.......................X...] duration: 1.952155201s FAIL (2025-07-28 00:32:09 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.95217552s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [.................................X...XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232779669s FAIL (2025-07-28 00:32:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 55m19.503647909s) [noise=true , address=104.36.82.11:4116 , piece_id=BUBZZJU3M7VHXKVLSU65RP7LF4CR7U2ZNUV7I7NJ2DBIOAJX6I7Q, seg_pos={0 0}, upload=81489]: Uploading piece... Jul 28 00:31:09.781496807 (84h55m19.504059857s) [noise=true , address=104.36.82.5:4850 , piece_id=GRA6TIREH63NIANSQABVYAWBLKNSIYO3I3NOQHO2OG6ZHMPEY5HA, seg_pos={0 0}, upload=81489]: Uploading piece... Jul 28 00:31:09.782085443 (84h55m19.504648483s) [noise=true , address=104.36.82.49:4990 , piece_id=EOYPY2SZKA6UZ66RNOTQSCLQQFUSPBPSV7ICWYHWRMIDMQ7SHNDQ, seg_pos={0 0}, upload=81489]: Uploading piece... Jul 28 00:31:09.782091945 (84h55m19.504654995s) [noise=true , address=104.36.82.12:4165 , piece_id=XCEP2UUHZ5RRH2GVL22VPMSFLKRJEGHT2I4CP5OPYS3V52C4NDCQ, seg_pos={0 0}, upload=81489]: Uploading piece... Jul 28 00:31:09.782324030 (84h55m19.50488707s) [noise=true , address=79.127.219.38:4001 , piece_id=RGTDTRMJMUTIFKVK2P47XPNVWKF2YYU22TV3B5LMRMRXVURMZ2KA, seg_pos={0 0}, upload=81489]: Uploading piece... Jul 28 00:31:09.782572886 (84h55m19.505135936s) [noise=true , address=104.36.82.21:4484 , piece_id=5NXUK4G5OLOOO2NW5XYYN7SES5C7HQ5CEFVBG6YS6GAG6JGPKURA, seg_pos={0 0}, upload=81489]: Uploading piece... Jul 28 00:31:09.782790152 (84h55m19.505353192s) [noise=true , address=104.36.82.3:4774 , piece_id=6VDWT6HHR7NKG5SVO5FSJGB7L2WY7NXBGBNRTDKUGHYZSKSWMIZA, seg_pos={0 0}, upload=81489]: Uploading piece... Jul 28 00:31:09.783024480 (84h55m19.50558752s) [noise=true , address=104.36.82.16:4307 , piece_id=Z3QSDZTU2GWIEEMBX5V3PDS5DXJ6KSDJSSBQZYPXMJ6SUVNY4GQQ, seg_pos={0 0}, upload=81489]: Uploading piece... Jul 28 00:31:11.331208188 (84h55m21.053771228s) [upload=81489]: Next returned nil segment Jul 28 00:31:11.331229709 (84h55m21.053792749s) [upload=81489]: Waiting for error group managing segments... Jul 28 00:31:11.774734496 (84h55m21.497297536s) [noise=true , address=79.127.219.38:4001 , piece_id=RGTDTRMJMUTIFKVK2P47XPNVWKF2YYU22TV3B5LMRMRXVURMZ2KA, seg_pos={0 0}, upload=81489]: Done uploading piece. err: <nil> Jul 28 00:32:08.681757995 (84h56m18.404321035s) [noise=true , address=104.36.82.16:4307 , piece_id=Z3QSDZTU2GWIEEMBX5V3PDS5DXJ6KSDJSSBQZYPXMJ6SUVNY4GQQ, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:12QGfzeTEErAaE5YRTLi1w7SZoJbo2xVKb1Sykj1pW5fbCZYSCn, address:104.36.82.16:4307): protocol: rpc: dial tcp 104.36.82.16:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.681833289 (84h56m18.404396329s) [noise=true , address=104.36.82.51:4050 , piece_id=EDYL3VAHLL4TFHQ6JAQALHLDVZOWELTRHPYW5VV6CKVUEQNJ222Q, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:104.36.82.51:4050): protocol: rpc: dial tcp 104.36.82.51:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.681883946 (84h56m18.404446986s) [noise=true , address=104.36.82.9:4008 , piece_id=NU4H6PC4C355SHYUBDEYAAJ2EKOUXZKTOFPBCFVGTRK7R2O5PFIQ, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.681916718 (84h56m18.404479758s) [noise=true , address=104.36.82.47:4540 , piece_id=56Z54VS2WTWVSIETQOYV3OA5WUSRRPEYTHLNZBZFVMJ4KEPRMC5Q, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:104.36.82.47:4540): protocol: rpc: dial tcp 104.36.82.47:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.681948459 (84h56m18.404511499s) [noise=true , address=104.36.82.49:4990 , piece_id=EOYPY2SZKA6UZ66RNOTQSCLQQFUSPBPSV7ICWYHWRMIDMQ7SHNDQ, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:1gDzCKcMaLTisHw68GYkMbcMJF7rbKnUt48etS8Q7KeiXedbU4, address:104.36.82.49:4990): protocol: rpc: dial tcp 104.36.82.49:4990: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682046897 (84h56m18.404609937s) [noise=true , address=104.36.82.12:4165 , piece_id=XCEP2UUHZ5RRH2GVL22VPMSFLKRJEGHT2I4CP5OPYS3V52C4NDCQ, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:13Ksy7YCGV5u3T5Z6WwKtMjFq6Rs4bayzR7mQFyXbSFuJjaH4F, address:104.36.82.12:4165): protocol: rpc: dial tcp 104.36.82.12:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682103395 (84h56m18.404666435s) [noise=true , address=104.36.82.52:4119 , piece_id=YMJMECNMEDBIFRBYLGEHJCCQADXWF5YBV2XRY6XLNNTANBKJQWHQ, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:12mTLjL6Y6VVjqztwVXYif4v7tyftjLafztLUibajksMHCFraFG, address:104.36.82.52:4119): protocol: rpc: dial tcp 104.36.82.52:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682167348 (84h56m18.404730378s) [noise=true , address=104.36.82.41:4786 , piece_id=F32Y2QH7M65BU7LCRRVGHOKWG76GE5O3IGSMSHHENGAJROCYS5GQ, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q, address:104.36.82.41:4786): protocol: rpc: dial tcp 104.36.82.41:4786: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682248964 (84h56m18.404812004s) [noise=true , address=104.36.82.54:4190 , piece_id=MVGURVFLZIPY2SKH73VRNZUNDZCO2J3BQMDCIHWALGKWG3NMJ7UA, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682294511 (84h56m18.404857551s) [noise=true , address=104.36.82.53:4145 , piece_id=PBPYMEVYJMECJLDVDOMBLQQE4SG43K2EJT4ZIMHHMWA6LNQES4TQ, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: write tcp 188.166.225.170:34646->104.36.82.53:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682334077 (84h56m18.404897117s) [noise=true , address=104.36.82.5:4850 , piece_id=GRA6TIREH63NIANSQABVYAWBLKNSIYO3I3NOQHO2OG6ZHMPEY5HA, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi, address:104.36.82.5:4850): protocol: rpc: dial tcp 104.36.82.5:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682369875 (84h56m18.404932915s) [noise=true , address=104.36.82.13:4212 , piece_id=ABLX7JJ7GJUBUBRPRPWWSVSVMZQQDLHQVC4A5IBEEWBBHAL75SMA, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682436473 (84h56m18.404999513s) [noise=true , address=104.36.82.10:4102 , piece_id=U6SX7XKPMSQLEFKXS4ULWXGPIXB3OZG3EMNHDHIWUOMXOMP2LNHA, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:1JEXRhcy1TphHqJXcd6qDv5gCdr6W2nJUcGoarZbsGt3TrtePy, address:104.36.82.10:4102): protocol: rpc: dial tcp 104.36.82.10:4102: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682464146 (84h56m18.405027186s) [noise=true , address=104.36.82.11:4116 , piece_id=BUBZZJU3M7VHXKVLSU65RP7LF4CR7U2ZNUV7I7NJ2DBIOAJX6I7Q, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:123aoo9g11aJP9hiL8shu2dx44tjyKzEWqPUjHUdXtR7LLFyng8, address:104.36.82.11:4116): protocol: rpc: dial tcp 104.36.82.11:4116: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682530663 (84h56m18.405093703s) [noise=true , address=104.36.82.21:4484 , piece_id=5NXUK4G5OLOOO2NW5XYYN7SES5C7HQ5CEFVBG6YS6GAG6JGPKURA, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: rpc: dial tcp 104.36.82.21:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682561802 (84h56m18.405137076s) [noise=true , address=104.36.82.3:4774 , piece_id=6VDWT6HHR7NKG5SVO5FSJGB7L2WY7NXBGBNRTDKUGHYZSKSWMIZA, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: rpc: dial tcp 104.36.82.3:4774: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682582472 (84h56m18.405145512s) [noise=true , address=104.36.82.50:4037 , piece_id=BEPAODIH6CG62BOTUCWMNWCHWAICSL4QNWSVIS6GKAXEWREQMBLQ, seg_pos={0 0}, upload=81489]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: write tcp 188.166.225.170:44124->104.36.82.50:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:32:08.682605506 (84h56m18.405168546s) [seg_pos={0 0}, upload=81489]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:32:08.913172440 (84h56m18.6357355s) [upload=81489]: Done uploading segments... Jul 28 00:32:08.913202026 (84h56m18.635765066s) [upload=81489]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.232795158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................X.......X.XX....X....X...] duration: 5.840965856s FAIL (2025-07-28 00:31:50 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 5.840992387s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X.X..........X.....X.X....] duration: 5.968349099s FAIL (2025-07-28 00:31:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.968374037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................X.......X.XX....X....X..] duration: 5.78484549s FAIL (2025-07-28 00:31:17 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 5.784876308s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [..................................X...XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.237756454s FAIL (2025-07-28 00:31:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== L5GLKLHAWZ3CP34LAMXZ5HQ7VM5FJINWQEZHKH63XRA, seg_pos={0 0}, upload=81473]: Uploading piece... Jul 28 00:30:09.315880511 (84h54m19.038443561s) [noise=true , address=104.36.82.20:4472 , piece_id=3NT6GH43DWZ6QAROTKN4GUTKBU54C4JY27MAZMDFMRGIP3ZFFPNA, seg_pos={0 0}, upload=81473]: Uploading piece... Jul 28 00:30:09.315886262 (84h54m19.038449302s) [noise=true , address=104.36.82.42:4819 , piece_id=SPVP5ZWCMYPRJEOV7NYBEPZTOU2F3LFYNCI2LGVSMXFX5Y7L4BGQ, seg_pos={0 0}, upload=81473]: Uploading piece... Jul 28 00:30:09.316124338 (84h54m19.038687378s) [noise=true , address=104.36.82.12:4179 , piece_id=6ONZVCL5TM6XXM2AS2YRSONW5G2PZX27525I74KV5YDYNXDISQOA, seg_pos={0 0}, upload=81473]: Uploading piece... Jul 28 00:30:09.316408992 (84h54m19.038972032s) [noise=true , address=104.36.82.6:4880 , piece_id=LSOYO37RTEZLBG5VOGS2PLBBA6EVA5FFOBOMJCN6VVGHZVCZACHQ, seg_pos={0 0}, upload=81473]: Uploading piece... Jul 28 00:30:09.316705359 (84h54m19.039268399s) [noise=true , address=104.36.82.14:4245 , piece_id=K7PKDJSD6QAKNPSKTCM237HAT4O2ZASB4DONSRT2XYNXQ6BKJTDQ, seg_pos={0 0}, upload=81473]: Uploading piece... Jul 28 00:30:09.317028257 (84h54m19.039591297s) [noise=true , address=104.36.82.52:4108 , piece_id=TJZHDA6PJLBGKZ7Y45MX2H37UFUGP6EBBT2Z463LVIN5GEVGJD4A, seg_pos={0 0}, upload=81473]: Uploading piece... Jul 28 00:30:10.811127937 (84h54m20.533690977s) [upload=81473]: Next returned nil segment Jul 28 00:30:10.811154317 (84h54m20.533717357s) [upload=81473]: Waiting for error group managing segments... Jul 28 00:30:11.323185866 (84h54m21.045748906s) [noise=true , address=79.127.201.215:4354 , piece_id=AVOXGPFUBRFWOLPNHPHICKP5TIMVCBO3OVYGI2GEKWBZOS676NBQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: <nil> Jul 28 00:31:08.214913727 (84h55m17.937476778s) [noise=true , address=104.36.82.42:4819 , piece_id=SPVP5ZWCMYPRJEOV7NYBEPZTOU2F3LFYNCI2LGVSMXFX5Y7L4BGQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:12MZhrDu1fjMN7XFoR2h1q3JPLHJYYURGQwQgcgLuucSrGShjoS, address:104.36.82.42:4819): protocol: write tcp 188.166.225.170:50914->104.36.82.42:4819: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.214967641 (84h55m17.937530681s) [noise=true , address=104.36.82.4:4825 , piece_id=ZWHDXCROQL5GLKLHAWZ3CP34LAMXZ5HQ7VM5FJINWQEZHKH63XRA, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:104.36.82.4:4825): protocol: write tcp 188.166.225.170:54644->104.36.82.4:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.214994943 (84h55m17.937557983s) [noise=true , address=104.36.82.12:4179 , piece_id=6ONZVCL5TM6XXM2AS2YRSONW5G2PZX27525I74KV5YDYNXDISQOA, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:129pP9f3rFfTvP6MhG894N7WHuJRZP8aWhzJf1zBwZ76wbzVbZa, address:104.36.82.12:4179): protocol: rpc: dial tcp 104.36.82.12:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215011113 (84h55m17.937574153s) [noise=true , address=104.36.82.54:4174 , piece_id=4BLMWJZIUUDDMVULQWLZZVPZWNCZEVZAYZYDD3VK56E2IHV5E6KA, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: write tcp 188.166.225.170:55018->104.36.82.54:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215050108 (84h55m17.937613148s) [noise=true , address=104.36.82.46:4514 , piece_id=NFMU6NF43S2ST53R64WRIBTOXZLUWAKDCYP5GWGLI4YEULW5DSQQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: write tcp 188.166.225.170:39740->104.36.82.46:4514: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215062872 (84h55m17.937625912s) [noise=true , address=104.36.82.14:4245 , piece_id=K7PKDJSD6QAKNPSKTCM237HAT4O2ZASB4DONSRT2XYNXQ6BKJTDQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: rpc: dial tcp 104.36.82.14:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215086518 (84h55m17.937649558s) [noise=true , address=104.36.82.20:4472 , piece_id=3NT6GH43DWZ6QAROTKN4GUTKBU54C4JY27MAZMDFMRGIP3ZFFPNA, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:12GACnovbrme3fQknSDpZWVLq838VKop9mPjxnVXGdJ8KaqyaTi, address:104.36.82.20:4472): protocol: write tcp 188.166.225.170:45758->104.36.82.20:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215100725 (84h55m17.937663765s) [noise=true , address=104.36.82.9:4029 , piece_id=HMQQPRLIQR72MNJCFSVGPFKWHKAU2EZEK6XKOKPUJGBHW5KWVCZQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:12snsP1y5fbxY8XrDoXihzQrsBxJ5Y38C2jSGhHtV1ukN426eiP, address:104.36.82.9:4029): protocol: rpc: dial tcp 104.36.82.9:4029: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215125703 (84h55m17.937688743s) [noise=true , address=104.36.82.52:4108 , piece_id=TJZHDA6PJLBGKZ7Y45MX2H37UFUGP6EBBT2Z463LVIN5GEVGJD4A, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: write tcp 188.166.225.170:38876->104.36.82.52:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215153136 (84h55m17.937716176s) [noise=true , address=104.36.82.15:4260 , piece_id=2U7ZZJD5PJYJDFMCYGFKP4W2PIVTHBOAQYYOBMANKLM7PXDYXYRA, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:1aLsgrV3k6vUSavgPkPGA2FF9HR3ryN9bnHutruFJHW1GYixeL, address:104.36.82.15:4260): protocol: rpc: dial tcp 104.36.82.15:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215164487 (84h55m17.937727527s) [noise=true , address=104.36.82.47:4532 , piece_id=MVPSNL2Z3Y6W5THFG5YCDJHNYQOAOEFEK7JC7HKGQRIK7UYXU4KQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: write tcp 188.166.225.170:47328->104.36.82.47:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215200516 (84h55m17.937763556s) [noise=true , address=104.36.82.5:4846 , piece_id=YJHNLSHG4AKFEXTWSSGJUREJRHHGOPOMBXCDHDDKHPMP76ITUMNA, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: write tcp 188.166.225.170:39548->104.36.82.5:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215203912 (84h55m17.937766952s) [noise=true , address=104.36.82.11:4141 , piece_id=LKU2GDUWJ4DF24GUF7N5L5HGPOXZEMJR2FO7HFJKSYV3COYMGMPQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:4141): protocol: rpc: dial tcp 104.36.82.11:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215231575 (84h55m17.937794615s) [noise=true , address=104.36.82.53:4145 , piece_id=BKBSWXXQIEH6GAO3PXKHTXKAP5RO4GI7VMWH2GHCQ3JLMTH6M2DQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: write tcp 188.166.225.170:34284->104.36.82.53:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215244410 (84h55m17.93780745s) [noise=true , address=104.36.82.41:4781 , piece_id=56MTISJ7SHJMD6SUQZJPVIQCLQVJRUK4WKPDIIYVYPNPGK6X36BA, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:1QhCkTzSogXFKLmFV4wQUcALhvm9YhCSQudLxSrvQWmepheiFi, address:104.36.82.41:4781): protocol: rpc: dial tcp 104.36.82.41:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215263938 (84h55m17.937826978s) [noise=true , address=104.36.82.40:4949 , piece_id=HN3YGGFI4SADN5VKNAJIOANBRB5NNZ6FHBNVUCTAURCX43H4V3QQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:104.36.82.40:4949): protocol: rpc: dial tcp 104.36.82.40:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215299706 (84h55m17.937862736s) [noise=true , address=104.36.82.6:4880 , piece_id=LSOYO37RTEZLBG5VOGS2PLBBA6EVA5FFOBOMJCN6VVGHZVCZACHQ, seg_pos={0 0}, upload=81473]: Done uploading piece. err: ecclient: upload failed (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW, address:104.36.82.6:4880): protocol: rpc: dial tcp 104.36.82.6:4880: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:08.215327409 (84h55m17.937890439s) [seg_pos={0 0}, upload=81473]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:31:08.450124376 (84h55m18.172687416s) [upload=81473]: Done uploading segments... Jul 28 00:31:08.450153682 (84h55m18.172716722s) [upload=81473]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.237771473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................X............X.......................X..] duration: 2.033520161s FAIL (2025-07-28 00:30:54 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 2.033546382s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................X.......X.XX....X....X.] duration: 5.902269129s FAIL (2025-07-28 00:30:46 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 5.902301412s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................X.X..........X.....X.X...] duration: 5.769486748s FAIL (2025-07-28 00:30:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.769518318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................X.X..........X.....X.X..] duration: 6.167570351s FAIL (2025-07-28 00:30:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.167598365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [...................................X...XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.247360731s FAIL (2025-07-28 00:30:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ise=true , address=104.36.82.46:4493 , piece_id=GZTBIMVTJOQWGFRAKQVEVE7H34K2GEQXLDC4LHM2XKVCIL7EQRIQ, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:08.887441785 (84h53m18.610004865s) [noise=true , address=104.36.82.54:4163 , piece_id=RWP2HS5JELWA3GYEPACV3XGK7LFMGKXZUDUMGIG2QHY2PQYS5YTA, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:08.887689739 (84h53m18.610252779s) [noise=true , address=104.36.82.43:4856 , piece_id=33SNVGLP2XTI7D6LBRSGYEBKLTLAYFDTIP3VFSKGGQ73QVU6L3ZA, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:08.887883930 (84h53m18.610446971s) [noise=true , address=104.36.82.7:4918 , piece_id=BVKNRJGYE3YGHY5M52Q6FWL5K4P7FU6N2UODEROC5LQG65QWMJMQ, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:08.888077161 (84h53m18.610640201s) [noise=true , address=104.36.82.3:4778 , piece_id=MKNARU5QZTMG625B4GFW4J4FNOLIZURB2HLL6GIOMDM6GAQTIX5Q, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:08.888259018 (84h53m18.610822059s) [noise=true , address=104.36.82.20:4464 , piece_id=IUT7RSUTOAUMW2BT4IN7LCYJTWPTQJVOHPA6QY2KKSKF4QZZYFCQ, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:08.888535427 (84h53m18.611098467s) [noise=true , address=104.36.82.22:4531 , piece_id=ZP3VXKPPV7F5LHDOK7WP6WG5OLYB2OEATDOIS66EGEILSPEKT4FA, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:08.888817667 (84h53m18.611380707s) [noise=true , address=104.36.82.14:4253 , piece_id=TDPWZKXMJBJAO362ABF6M2LVRCVVLDXXBELY2HAF27ANJWKQKFRQ, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:08.889002441 (84h53m18.611565481s) [noise=true , address=104.36.82.4:4836 , piece_id=BFT53MZPK5UKFOK7EV5ZLXU3PBLG6CYR5AUSTGFNW7WLMQGMKTPA, seg_pos={0 0}, upload=81459]: Uploading piece... Jul 28 00:29:10.028440866 (84h53m19.751003906s) [upload=81459]: Next returned nil segment Jul 28 00:29:10.028461976 (84h53m19.751025026s) [upload=81459]: Waiting for error group managing segments... Jul 28 00:29:10.317759258 (84h53m20.040322308s) [noise=true , address=79.127.205.233:4656 , piece_id=6JFAP3LPIDO6OQXRRUERLLFXJONIQLBE46NKC6UH4NRKTBN3IJKA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: <nil> Jul 28 00:29:14.415848674 (84h53m24.138411714s) [noise=true , address=79.127.201.215:4350 , piece_id=IZS64LLQDU34BQTMIY7X4HR4FKW5X5TMRAOXSQ4W3GBZNMOB74AQ, seg_pos={0 0}, upload=81459]: Done uploading piece. err: <nil> Jul 28 00:30:07.739033102 (84h54m17.461596142s) [noise=true , address=104.36.82.54:4163 , piece_id=RWP2HS5JELWA3GYEPACV3XGK7LFMGKXZUDUMGIG2QHY2PQYS5YTA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: write tcp 188.166.225.170:50186->104.36.82.54:4163: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739408259 (84h54m17.461971299s) [noise=true , address=104.36.82.22:4531 , piece_id=ZP3VXKPPV7F5LHDOK7WP6WG5OLYB2OEATDOIS66EGEILSPEKT4FA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12jGenn3Fh2VB36jVPzyy2LxJgbLraXxL2yobw9tsBJLSDPJycN, address:104.36.82.22:4531): protocol: rpc: dial tcp 104.36.82.22:4531: i/o timeout; conn closed: op Jul 28 00:30:07.739451262 (84h54m17.462014302s) [noise=true , address=104.36.82.46:4493 , piece_id=GZTBIMVTJOQWGFRAKQVEVE7H34K2GEQXLDC4LHM2XKVCIL7EQRIQ, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 188.166.225.170:59864->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739489314 (84h54m17.462052354s) [noise=true , address=104.36.82.52:4103 , piece_id=P4ZFDPEO6SGYRVKVNDINKSRNMEUMTSOTOFBAYVLBQOC3KSBWYXAA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: rpc: dial tcp 104.36.82.52:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739535352 (84h54m17.462098393s) [noise=true , address=104.36.82.40:4950 , piece_id=ANGPYTWM4DOD7CEE63BFI2RHWQEI5KJJHCDUSEFQAHJOPEICY4WQ, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk, address:104.36.82.40:4950): protocol: rpc: dial tcp 104.36.82.40:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739565450 (84h54m17.46212849s) [noise=true , address=104.36.82.3:4778 , piece_id=MKNARU5QZTMG625B4GFW4J4FNOLIZURB2HLL6GIOMDM6GAQTIX5Q, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12JUnoFQa1s1amHUHWiTtQVatCzgMHii1kskRjm9DYjkEuke3Ge, address:104.36.82.3:4778): protocol: rpc: dial tcp 104.36.82.3:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739613392 (84h54m17.462176432s) [noise=true , address=104.36.82.4:4836 , piece_id=BFT53MZPK5UKFOK7EV5ZLXU3PBLG6CYR5AUSTGFNW7WLMQGMKTPA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:104.36.82.4:4836): protocol: rpc: dial tcp 104.36.82.4:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739646585 (84h54m17.462209625s) [noise=true , address=104.36.82.53:4147 , piece_id=BVGHLS63MBLO2EMSNIFICVH6LY7GXOEDE7G4JE4GY5E7Y5EXEWPQ, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 188.166.225.170:34608->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739669419 (84h54m17.462232459s) [noise=true , address=104.36.82.42:4828 , piece_id=6WPO7MGZONWFDKUEFVLWNXVE4SWJ7B6GXZBMRXYIDZJT5BOF4IAA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:1cX7FtKC8KMopnX9zvkT6fYdnutK84cFjLW6xMwpXuGsqpiN8k, address:104.36.82.42:4828): protocol: write tcp 188.166.225.170:60528->104.36.82.42:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739715738 (84h54m17.462278778s) [noise=true , address=104.36.82.56:4261 , piece_id=3CYQYT4IOKLOOZCIPWINDAQWB7MDM5VVSGAUJ76OB46KC3DYPUPQ, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: rpc: dial tcp 104.36.82.56:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739759511 (84h54m17.462322551s) [noise=true , address=104.36.82.44:4872 , piece_id=FEZYYYVMNTD73OIB5OWIULSMIHUIRU3L7QQ6OIYCL5DZNDYJJQEA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:104.36.82.44:4872): protocol: rpc: dial tcp 104.36.82.44:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739795361 (84h54m17.462358401s) [noise=true , address=104.36.82.7:4918 , piece_id=BVKNRJGYE3YGHY5M52Q6FWL5K4P7FU6N2UODEROC5LQG65QWMJMQ, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: rpc: dial tcp 104.36.82.7:4918: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739827522 (84h54m17.462390562s) [noise=true , address=104.36.82.20:4464 , piece_id=IUT7RSUTOAUMW2BT4IN7LCYJTWPTQJVOHPA6QY2KKSKF4QZZYFCQ, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:1xKikZTtJ6dr1ry8oAkLc7fmyB51jydk87BWoxuTTzKFDeToEY, address:104.36.82.20:4464): protocol: rpc: dial tcp 104.36.82.20:4464: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739834094 (84h54m17.462397134s) [noise=true , address=104.36.82.14:4253 , piece_id=TDPWZKXMJBJAO362ABF6M2LVRCVVLDXXBELY2HAF27ANJWKQKFRQ, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14, address:104.36.82.14:4253): protocol: rpc: dial tcp 104.36.82.14:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739861156 (84h54m17.462424196s) [noise=true , address=104.36.82.55:4207 , piece_id=PVVAD3TFTA6TAHCBQQN4GT3HPBVJEMGOCJKRXOMG2KKLH3AJ2DEA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739883058 (84h54m17.462446098s) [noise=true , address=104.36.82.43:4856 , piece_id=33SNVGLP2XTI7D6LBRSGYEBKLTLAYFDTIP3VFSKGGQ73QVU6L3ZA, seg_pos={0 0}, upload=81459]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:07.739908477 (84h54m17.462471517s) [seg_pos={0 0}, upload=81459]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:30:07.984812080 (84h54m17.70737513s) [upload=81459]: Done uploading segments... Jul 28 00:30:07.984846245 (84h54m17.707409285s) [upload=81459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.247388394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................X............X.......................X.] duration: 1.905960721s FAIL (2025-07-28 00:30:04 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.905999696s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................X.......X.XX....X....X] duration: 5.634699794s FAIL (2025-07-28 00:29:45 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 5.634725233s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X.X..........X.....X.X.] duration: 5.732139311s FAIL (2025-07-28 00:29:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.732166926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................X.X..........X.....X.X] duration: 30.236993658s FAIL (2025-07-28 00:29:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.237015551s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................X............X.......................X] duration: 1.606808783s FAIL (2025-07-28 00:29:02 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.606830686s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [....................................X...XXXXXXXXXXXXXXXXXXXX] duration: 1m0.234042979s FAIL (2025-07-28 00:29:00 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== load=81442]: Uploading piece... Jul 28 00:28:01.715304911 (84h52m11.437867951s) [noise=true , address=104.36.82.42:4820 , piece_id=CCFMFB42I6ZBJUJEJCDTKOSS5VB6IEJTNGIDZAVLCLLGQZFKLEKA, seg_pos={0 0}, upload=81442]: Uploading piece... Jul 28 00:28:01.715487180 (84h52m11.43805023s) [noise=true , address=104.36.82.52:4101 , piece_id=QKMOSYYUS2QQHXB5NBAI4K33XMJSRQBW4R6NLW2BLLPJU2ORFZWQ, seg_pos={0 0}, upload=81442]: Uploading piece... Jul 28 00:28:01.715521917 (84h52m11.438084957s) [noise=true , address=104.36.82.56:4252 , piece_id=FC4W55PLKPZKNDRSPDOAIX3MX7KEHHUWRJ6HCYIVD7HLAPJMWYQQ, seg_pos={0 0}, upload=81442]: Uploading piece... Jul 28 00:28:01.715743971 (84h52m11.438307011s) [noise=true , address=104.36.82.42:4815 , piece_id=JNOLUULYTG4ZVZ5AGWPBHQ5JYZGL6APUYLO2Q2TWOAWKP6FN5HQQ, seg_pos={0 0}, upload=81442]: Uploading piece... Jul 28 00:28:01.715752237 (84h52m11.438315277s) [noise=true , address=104.36.82.41:4774 , piece_id=YDBCHOPRX4AQMMISLUTXTEMWPYD7GCZ4XAGZO3DKW3HWEDDE34XQ, seg_pos={0 0}, upload=81442]: Uploading piece... Jul 28 00:28:01.715963111 (84h52m11.438526151s) [noise=true , address=104.36.82.3:4781 , piece_id=I2IS6MHRJAGH4WO2IDIESYIWODZKJAUUS6FAZMQ4NIMNYDI4RSWQ, seg_pos={0 0}, upload=81442]: Uploading piece... Jul 28 00:28:01.716050508 (84h52m11.438613558s) [noise=true , address=104.36.82.47:4532 , piece_id=FOHFBJ3JCJ5CKSZ74OOCT24BD7CHCDOE65HJNQWVHFUHBS2ETSIQ, seg_pos={0 0}, upload=81442]: Uploading piece... Jul 28 00:28:01.716310566 (84h52m11.438873616s) [noise=true , address=104.36.82.11:4139 , piece_id=VF7GOUAAQVAUYPWGJEDNDUBMCSZPW6Z6MBRM3YKZGD2PWN2C6UOA, seg_pos={0 0}, upload=81442]: Uploading piece... Jul 28 00:29:00.547854540 (84h53m10.27041758s) [upload=81442]: Next segment err: context deadline exceeded Jul 28 00:29:00.549016414 (84h53m10.271579454s) [noise=true , address=104.36.82.42:4815 , piece_id=JNOLUULYTG4ZVZ5AGWPBHQ5JYZGL6APUYLO2Q2TWOAWKP6FN5HQQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1F59CEsZMSXL8DibXEqG6XQKirqjEfYW8UhugnFCPnUEUWLHDX, address:104.36.82.42:4815): protocol: rpc: dial tcp 104.36.82.42:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549044748 (84h53m10.271607788s) [noise=true , address=104.36.82.14:4251 , piece_id=PYAOCLZIAZKDYKFOA52AKLVW7KFI224A2CS5WAIQH7LLLRRANZGQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549074795 (84h53m10.271637835s) [noise=true , address=104.36.82.41:4774 , piece_id=YDBCHOPRX4AQMMISLUTXTEMWPYD7GCZ4XAGZO3DKW3HWEDDE34XQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb, address:104.36.82.41:4774): protocol: rpc: dial tcp 104.36.82.41:4774: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549116354 (84h53m10.271679394s) [noise=true , address=104.36.82.56:4252 , piece_id=FC4W55PLKPZKNDRSPDOAIX3MX7KEHHUWRJ6HCYIVD7HLAPJMWYQQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1UDFNiqXb4riRuewJDPrKFykVWLiryNKqgkH6uCZszz6Sr8naB, address:104.36.82.56:4252): protocol: rpc: dial tcp 104.36.82.56:4252: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549131865 (84h53m10.271694905s) [noise=true , address=104.36.82.3:4781 , piece_id=I2IS6MHRJAGH4WO2IDIESYIWODZKJAUUS6FAZMQ4NIMNYDI4RSWQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:104.36.82.3:4781): protocol: rpc: dial tcp 104.36.82.3:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549155440 (84h53m10.27171848s) [noise=true , address=104.36.82.20:4458 , piece_id=MEKHH7JFY4NXFP4OUQJECB5HS3KRUN4SAQ43LPKGABYPCII5DSOA, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:12SEsKJVwuQV1Z6s6doBrqsST1KhqbBREjwFYeEpME7gHExaG5d, address:104.36.82.20:4458): protocol: rpc: dial tcp 104.36.82.20:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549208071 (84h53m10.271771111s) [noise=true , address=104.36.82.11:4139 , piece_id=VF7GOUAAQVAUYPWGJEDNDUBMCSZPW6Z6MBRM3YKZGD2PWN2C6UOA, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: rpc: dial tcp 104.36.82.11:4139: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549216937 (84h53m10.271779967s) [noise=true , address=104.36.82.46:4515 , piece_id=PGGPFVZHMOMBHP7CHPREHQJVUZ66RMELJBFFXBQYRZAMCGF7M5UA, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: rpc: dial tcp 104.36.82.46:4515: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549247335 (84h53m10.271810375s) [noise=true , address=104.36.82.44:4893 , piece_id=H6W74NMNU427ENMB5U46FUHEEMQD2DS4QZLORGWHEYMZYHADDAUA, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: rpc: dial tcp 104.36.82.44:4893: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549285458 (84h53m10.271848498s) [noise=true , address=104.36.82.21:4477 , piece_id=MA6PTPUA3RHZAWNKJNFYG77D7Y4IJTK7YQPFNND7SAMQNQIHHRZA, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY, address:104.36.82.21:4477): protocol: rpc: dial tcp 104.36.82.21:4477: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549315335 (84h53m10.271878375s) [noise=true , address=104.36.82.51:4079 , piece_id=G6YM5XVCZTO2XNZWFAPQPVKPVLDXQ4F7SDNH5Y6Y635PFK5F2JIQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y, address:104.36.82.51:4079): protocol: rpc: dial tcp 104.36.82.51:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549329392 (84h53m10.271892432s) [noise=true , address=104.36.82.52:4101 , piece_id=QKMOSYYUS2QQHXB5NBAI4K33XMJSRQBW4R6NLW2BLLPJU2ORFZWQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549368747 (84h53m10.271931787s) [noise=true , address=104.36.82.42:4820 , piece_id=CCFMFB42I6ZBJUJEJCDTKOSS5VB6IEJTNGIDZAVLCLLGQZFKLEKA, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh, address:104.36.82.42:4820): protocol: rpc: dial tcp 104.36.82.42:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549382744 (84h53m10.271945774s) [noise=true , address=104.36.82.50:4045 , piece_id=RZHH5RSDBNSCHZB7ZLG7SXLPMEXWZM5KR4X6JUCK4TOSNRTO6GVQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:104.36.82.50:4045): protocol: rpc: dial tcp 104.36.82.50:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549415717 (84h53m10.271978757s) [noise=true , address=104.36.82.54:4188 , piece_id=VLHOGJFCZVI7ZY7JQEPW2S2EFBZVCZEZPEIMFL4UP3R62PZDQD4Q, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG, address:104.36.82.54:4188): protocol: write tcp 188.166.225.170:54936->104.36.82.54:4188: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549468719 (84h53m10.272031749s) [noise=true , address=104.36.82.10:4089 , piece_id=OLEQUNY4QRABHKMACDVA5UTIIDOQDINC2CSBR7MP5724LGQVD4NQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 188.166.225.170:38556->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549477255 (84h53m10.272040295s) [noise=true , address=104.36.82.47:4532 , piece_id=FOHFBJ3JCJ5CKSZ74OOCT24BD7CHCDOE65HJNQWVHFUHBS2ETSIQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: write tcp 188.166.225.170:47522->104.36.82.47:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549485992 (84h53m10.272049032s) [noise=true , address=104.36.82.53:4137 , piece_id=FCWMACIPJZAODEWSE6DOAH3XFKRJNRXVE6OQZFHP557JEQPOLJWQ, seg_pos={0 0}, upload=81442]: Done uploading piece. err: ecclient: upload failed (node:1dZZBaTDA1kbdJmNZzzYgdPkKE2rNvEqdWaMGUxuTD2ERbSanc, address:104.36.82.53:4137): protocol: rpc: dial tcp 104.36.82.53:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:00.549496772 (84h53m10.272059812s) [seg_pos={0 0}, upload=81442]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.234058308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X.X..........X.....X.] duration: 6.007484627s FAIL (2025-07-28 00:28:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 81424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: rpc: dial tcp 104.36.82.6:4874: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:00.887031015 (84h51m10.609594045s) [seg_pos={0 0}, upload=81424]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:27:00.887091471 (84h51m10.609654511s) [upload=81424]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:27:00.887096300 (84h51m10.60965934s) [upload=81424]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:27:01.163946963 (84h51m10.886510003s) [upload=81424]: Done uploading segments... Jul 28 00:27:01.163965929 (84h51m10.886528969s) [upload=81424]: Upload finished. err: <nil> Jul 28 00:27:19.597171719 (84h51m29.319734759s) [upload=81431]: Starting upload Jul 28 00:27:19.597272533 (84h51m29.319835573s) [upload=81431]: Done starting upload Jul 28 00:27:19.597365821 (84h51m29.319928861s) [upload=81431]: Uploading segments... Jul 28 00:27:19.597700883 (84h51m29.320263923s) [upload=81431]: Got next segment. Inline: false Jul 28 00:27:19.597736120 (84h51m29.32029915s) [upload=81431]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:27:19.597741259 (84h51m29.320304299s) [upload=81431]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:27:20.045867384 (84h51m29.768430424s) [seg_pos={0 0}, upload=81431]: Begin upload segment... Jul 28 00:27:20.045927990 (84h51m29.76849103s) [seg_pos={0 0}, upload=81431]: Done begin upload segment. Jul 28 00:27:20.045958679 (84h51m29.768521729s) [noise=true , address=77.90.31.15:4698 , piece_id=X5L3W67HMYGRJVZYU72WUJ2ZT3VZ23DRBQJCNUJUV6YZCOL5ICTQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.045965942 (84h51m29.768528982s) [noise=true , address=77.90.31.13:4627 , piece_id=7NR6V4GBJR5WOHQRPDO4VIWT5BPEHWJMRE36UHKIIWSI6VQE6TAA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046055715 (84h51m29.768618755s) [noise=true , address=45.91.116.8:4047 , piece_id=GEDNNAD5CKPNTE7I2X2IK3SCI2G2F3JPKEVXW3HLIO44XOFKIURA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046336261 (84h51m29.768899302s) [noise=true , address=77.90.31.10:4519 , piece_id=N5HFTGGOKYCT42TILLRRHTJRYH4ZRWDQ32ADKNPGJN3ND2NQSUMQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046371579 (84h51m29.768934619s) [noise=true , address=45.91.116.9:4262 , piece_id=6GCZ7XGI4ZRYXBSN3MKUXSOKDFBBMKWWOXDU4VVU56ZDILYUHDEA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046630915 (84h51m29.769193955s) [noise=true , address=45.91.116.7:4010 , piece_id=FQ35UKX2QBOJ7KNNL44QQPNJWYOPBVUKNC67BTPFVQGJ2YUM2CSA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046655041 (84h51m29.769218081s) [noise=true , address=104.36.82.2:4731 , piece_id=KO3XYSDCEOM2YFS3LJOF27UIUULA4ZVRUXFWGPJYR5RJR3BCK45Q, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046905961 (84h51m29.769469001s) [noise=true , address=45.91.116.11:4118 , piece_id=5OBIONSVURIRITMD2HAVHLZTIDIBQOZJENJPKAI5JVKX6NSVG4ZQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046926390 (84h51m29.76948943s) [noise=true , address=77.90.31.16:4734 , piece_id=LDBACEVIC452LRFQ4NZWTWVJDIQZWD6HJL4XCDS55CJ362LACQBA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047221144 (84h51m29.769784184s) [noise=true , address=104.36.82.10:4077 , piece_id=RTY63EEKUWRHDL6S5WUDY5HAGUFEGFR5BWZO2QAKCBD7UC2JCQRQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047224911 (84h51m29.769787951s) [noise=true , address=104.36.82.4:4802 , piece_id=FFNXZCMDTS632HPHFDWEOXRR3XIMFDG2UEYOAXFFESCQJIWBRWUA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047435665 (84h51m29.769998705s) [noise=true , address=77.90.31.19:4842 , piece_id=SOQDD53C55RO67Q2C5HAYFTSKZP7ZMEWU42OON3YW24MQUMFNJ3A, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047460853 (84h51m29.770023893s) [noise=true , address=77.90.31.11:4555 , piece_id=SZQIQFGZUB225EUHDDT4JMJYMRIJ6FE25BXHM4ZGPDJM4SGTBPQQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047706483 (84h51m29.770269533s) [noise=true , address=77.90.31.21:4879 , piece_id=QBLWJK4D3WXQ4CO7U2XPX262ZDFPLLQKWUUUD43O5BHBCSBZBJLA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047961571 (84h51m29.770524611s) [noise=true , address=77.90.31.14:4662 , piece_id=PUBSU5WGV2VSPXER4D255XNPMZPBTLOLA5JS5ZBJFJWKWA6CYFBA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.048175260 (84h51m29.7707383s) [noise=true , address=104.36.82.6:4876 , piece_id=3HCF5CQHV7P73Z5I5NVFRUVGPBT5SCU24QXZGOZHRYG3MSA5HOUQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:21.770812904 (84h51m31.493375944s) [upload=81431]: Next returned nil segment Jul 28 00:27:21.770831569 (84h51m31.493394609s) [upload=81431]: Waiting for error group managing segments... Jul 28 00:27:22.047721888 (84h51m31.770284928s) [noise=true , address=45.91.116.9:4262 , piece_id=6GCZ7XGI4ZRYXBSN3MKUXSOKDFBBMKWWOXDU4VVU56ZDILYUHDEA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.069919327 (84h51m31.792482367s) [noise=true , address=77.90.31.14:4662 , piece_id=PUBSU5WGV2VSPXER4D255XNPMZPBTLOLA5JS5ZBJFJWKWA6CYFBA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.073840198 (84h51m31.796403248s) [noise=true , address=77.90.31.19:4842 , piece_id=SOQDD53C55RO67Q2C5HAYFTSKZP7ZMEWU42OON3YW24MQUMFNJ3A, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.074205587 (84h51m31.796768637s) [noise=true , address=45.91.116.11:4118 , piece_id=5OBIONSVURIRITMD2HAVHLZTIDIBQOZJENJPKAI5JVKX6NSVG4ZQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.142825163 (84h51m31.865388203s) [noise=true , address=45.91.116.8:4047 , piece_id=GEDNNAD5CKPNTE7I2X2IK3SCI2G2F3JPKEVXW3HLIO44XOFKIURA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.144458218 (84h51m31.867021258s) [noise=true , address=77.90.31.16:4734 , piece_id=LDBACEVIC452LRFQ4NZWTWVJDIQZWD6HJL4XCDS55CJ362LACQBA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.172639546 (84h51m31.895202586s) [noise=true , address=77.90.31.10:4519 , piece_id=N5HFTGGOKYCT42TILLRRHTJRYH4ZRWDQ32ADKNPGJN3ND2NQSUMQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.218333209 (84h51m31.940896249s) [noise=true , address=77.90.31.13:4627 , piece_id=7NR6V4GBJR5WOHQRPDO4VIWT5BPEHWJMRE36UHKIIWSI6VQE6TAA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.223604784 (84h51m31.946167824s) [noise=true , address=77.90.31.21:4879 , piece_id=QBLWJK4D3WXQ4CO7U2XPX262ZDFPLLQKWUUUD43O5BHBCSBZBJLA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.236036920 (84h51m31.958599961s) [noise=true , address=77.90.31.15:4698 , piece_id=X5L3W67HMYGRJVZYU72WUJ2ZT3VZ23DRBQJCNUJUV6YZCOL5ICTQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.251041218 (84h51m31.973604258s) [noise=true , address=45.91.116.7:4010 , piece_id=FQ35UKX2QBOJ7KNNL44QQPNJWYOPBVUKNC67BTPFVQGJ2YUM2CSA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.279611289 (84h51m32.002174329s) [noise=true , address=77.90.31.11:4555 , piece_id=SZQIQFGZUB225EUHDDT4JMJYMRIJ6FE25BXHM4ZGPDJM4SGTBPQQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:48.947845071 (84h51m58.670408111s) [noise=true , address=104.36.82.10:4077 , piece_id=RTY63EEKUWRHDL6S5WUDY5HAGUFEGFR5BWZO2QAKCBD7UC2JCQRQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:48.947912730 (84h51m58.67047577s) [noise=true , address=104.36.82.4:4802 , piece_id=FFNXZCMDTS632HPHFDWEOXRR3XIMFDG2UEYOAXFFESCQJIWBRWUA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:57482->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:48.947943999 (84h51m58.670507039s) [noise=true , address=104.36.82.6:4876 , piece_id=3HCF5CQHV7P73Z5I5NVFRUVGPBT5SCU24QXZGOZHRYG3MSA5HOUQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:51704->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:48.948013092 (84h51m58.670576132s) [noise=true , address=104.36.82.2:4731 , piece_id=KO3XYSDCEOM2YFS3LJOF27UIUULA4ZVRUXFWGPJYR5RJR3BCK45Q, seg_pos={0 0}, upload=81431]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:36822->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:48.948060071 (84h51m58.670623121s) [seg_pos={0 0}, upload=81431]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:27:49.184974758 (84h51m58.907537808s) [upload=81431]: Done uploading segments... Jul 28 00:27:49.185004114 (84h51m58.907567154s) [upload=81431]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.007511139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X.......X.XX....X....] duration: 5.754663095s FAIL (2025-07-28 00:28:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.754689966s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X............X.......................] duration: 3.220710769s FAIL (2025-07-28 00:28:16 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 3.220736257s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.....................................X...XXXXXXXXXXXXXXXXXXX] duration: 1m0.23564874s FAIL (2025-07-28 00:28:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== , piece_id=ZZOMN6DLLFKFRSRQKENMKZDCOCK3OJDZ5WBCXH22TDNVPGFXHDXQ, seg_pos={0 0}, upload=81427]: Uploading piece... Jul 28 00:27:01.178766486 (84h51m10.901329526s) [noise=true , address=104.36.82.55:4205 , piece_id=AN4Q4QQRUYUBJXGEV2SNURDSAKUDSCDTJLHCDNFLIBRKK65JCGVQ, seg_pos={0 0}, upload=81427]: Uploading piece... Jul 28 00:27:01.178970126 (84h51m10.901533166s) [noise=true , address=79.127.220.101:4057 , piece_id=W5KGAMCUAZSXA4CKXSM3KVFZ2QPDDAK2R4P4DUEA3SAF7Z3BUENQ, seg_pos={0 0}, upload=81427]: Uploading piece... Jul 28 00:27:01.179164438 (84h51m10.901727478s) [noise=true , address=104.36.82.10:4098 , piece_id=6QY3A5CYNQHYAJTC4XRO3N5KIGHJJ3WIW4FNLPEDV3KGNTVPGSUQ, seg_pos={0 0}, upload=81427]: Uploading piece... Jul 28 00:27:01.179366163 (84h51m10.901929203s) [noise=true , address=104.36.82.16:4311 , piece_id=QQEVGXOWS32WZERGFYZY5Y3XRUAL5LY3RSKMXBAEH6YQFX2TISDA, seg_pos={0 0}, upload=81427]: Uploading piece... Jul 28 00:27:01.179557760 (84h51m10.9021208s) [noise=true , address=104.36.82.47:4532 , piece_id=LWHWDYAYV7XR6YW33WDGKUWEOXELRDKVCWF54CXJBU2SEZFCJAXQ, seg_pos={0 0}, upload=81427]: Uploading piece... Jul 28 00:27:01.179748986 (84h51m10.902312026s) [noise=true , address=104.36.82.9:4031 , piece_id=Q23J7OAP3WKQJLBXXJNWKF7EIPJJ7FRV5VEDRFX7SMBK3QQ47F6A, seg_pos={0 0}, upload=81427]: Uploading piece... Jul 28 00:27:01.179977713 (84h51m10.902540753s) [noise=true , address=79.127.201.215:4344 , piece_id=4ROUDHRWBVX5447RNSXAWND7BIZTUMES3GUNBQ7Y6OIPHHWAJ2CA, seg_pos={0 0}, upload=81427]: Uploading piece... Jul 28 00:27:02.301980064 (84h51m12.024543114s) [upload=81427]: Next returned nil segment Jul 28 00:27:02.302002347 (84h51m12.024565447s) [upload=81427]: Waiting for error group managing segments... Jul 28 00:27:02.623564182 (84h51m12.346127222s) [noise=true , address=79.127.205.235:4687 , piece_id=VXRIQHM2BDENHTZK6FWTSYYFEUX64XKJXKCBZAJHRWMAWGBYOODQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: <nil> Jul 28 00:27:03.864066416 (84h51m13.586629456s) [noise=true , address=79.127.220.101:4057 , piece_id=W5KGAMCUAZSXA4CKXSM3KVFZ2QPDDAK2R4P4DUEA3SAF7Z3BUENQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: <nil> Jul 28 00:27:04.571810733 (84h51m14.294373773s) [noise=true , address=79.127.201.215:4344 , piece_id=4ROUDHRWBVX5447RNSXAWND7BIZTUMES3GUNBQ7Y6OIPHHWAJ2CA, seg_pos={0 0}, upload=81427]: Done uploading piece. err: <nil> Jul 28 00:28:00.088700472 (84h52m9.811263502s) [noise=true , address=104.36.82.5:4858 , piece_id=TLOCDKTLKARERYS37M5HFSWTKXBAK22IHWQY7EHPIDWYZ52DBBIQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:12PasA9NPExHCQgMSK1E1kz2HHkpXQmWq7LmrwY6sRmf22avri, address:104.36.82.5:4858): protocol: write tcp 188.166.225.170:51766->104.36.82.5:4858: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.088903270 (84h52m9.81146632s) [noise=true , address=104.36.82.11:4132 , piece_id=FPTTAKK5B4F5GJT4FXQV2GLLWLJBFR3TUSUBUYZCPGCMKW5BH4KA, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: write tcp 188.166.225.170:43784->104.36.82.11:4132: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.088951262 (84h52m9.811514292s) [noise=true , address=104.36.82.12:4163 , piece_id=3CMSMM275CJLBIVSITZPRU6LTZXLICQVWA7SLI5ZQL6MRUHL7RQQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:104.36.82.12:4163): protocol: write tcp 188.166.225.170:45126->104.36.82.12:4163: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.088988393 (84h52m9.811551423s) [noise=true , address=104.36.82.13:4207 , piece_id=35JMK3NNCWKKHGIAKS2XDFU6LPSNGXPOOW5VB4JSRZ6FFMJVH3HA, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: write tcp 188.166.225.170:45382->104.36.82.13:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089019653 (84h52m9.811582693s) [noise=true , address=104.36.82.41:4789 , piece_id=RGB5FE2KBXLOAJU73WWYB7W6PYQECME5A5CACZEDTI4AKRFWQ65A, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:1AXrW3TejqoDouALeZ1ddt8NqHZ5NZgZk5piqZHQNWgRjPHNqN, address:104.36.82.41:4789): protocol: write tcp 188.166.225.170:55990->104.36.82.41:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089116829 (84h52m9.811679869s) [noise=true , address=104.36.82.55:4205 , piece_id=AN4Q4QQRUYUBJXGEV2SNURDSAKUDSCDTJLHCDNFLIBRKK65JCGVQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ, address:104.36.82.55:4205): protocol: write tcp 188.166.225.170:42382->104.36.82.55:4205: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089155163 (84h52m9.811718203s) [noise=true , address=104.36.82.16:4311 , piece_id=QQEVGXOWS32WZERGFYZY5Y3XRUAL5LY3RSKMXBAEH6YQFX2TISDA, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:12qPZjuPpHwR2HwuBRK3F1QjN5zdU7StDPtN2PZ1ZeCP1ebqYmH, address:104.36.82.16:4311): protocol: write tcp 188.166.225.170:46200->104.36.82.16:4311: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089196141 (84h52m9.811759191s) [noise=true , address=104.36.82.56:4233 , piece_id=CEFMOX4O6ABDIFRTIK25OLJ5CMHWDPTHZU7B43TITY6MLLZLTDMQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: write tcp 188.166.225.170:57212->104.36.82.56:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089234534 (84h52m9.811797574s) [noise=true , address=104.36.82.42:4805 , piece_id=KICHUYBXWYS4DENCWFKP6BQ5I2E32YHSO3HZWATPFFBGHCVTOELA, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:1nh6Kih617STw2M8iHVjs4xP1HjjckJHyUxQ5z6ZFRhWwG6RXi, address:104.36.82.42:4805): protocol: write tcp 188.166.225.170:44014->104.36.82.42:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089286253 (84h52m9.811849293s) [noise=true , address=104.36.82.53:4136 , piece_id=TPCD7QDS37ZA6VMN63FRUVRDRI6HRDJTQ57HN6AKU6CRC543WCWQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:104.36.82.53:4136): protocol: write tcp 188.166.225.170:52292->104.36.82.53:4136: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089322112 (84h52m9.811885152s) [noise=true , address=104.36.82.14:4230 , piece_id=ZZOMN6DLLFKFRSRQKENMKZDCOCK3OJDZ5WBCXH22TDNVPGFXHDXQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:15hXnJMgf7FAJJh5y1S35eNLdvvqQR4yFot9M8FAmwJDyYVsWf, address:104.36.82.14:4230): protocol: write tcp 188.166.225.170:43852->104.36.82.14:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089349374 (84h52m9.811912404s) [noise=true , address=104.36.82.9:4031 , piece_id=Q23J7OAP3WKQJLBXXJNWKF7EIPJJ7FRV5VEDRFX7SMBK3QQ47F6A, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:104.36.82.9:4031): protocol: write tcp 188.166.225.170:55528->104.36.82.9:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089358161 (84h52m9.811921201s) [noise=true , address=104.36.82.10:4098 , piece_id=6QY3A5CYNQHYAJTC4XRO3N5KIGHJJ3WIW4FNLPEDV3KGNTVPGSUQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:59774->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089379933 (84h52m9.811942973s) [noise=true , address=104.36.82.3:4792 , piece_id=XTJXVQXAKUUPOPFTXBOCS6HDFG7Q5LFH4VXLVOQPO3J3ED2CYMVA, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:1fokaL7frEkFdVZRGmqaq3TmZVNYpt4agWCJZ8ABHfsYwkjAuv, address:104.36.82.3:4792): protocol: write tcp 188.166.225.170:55308->104.36.82.3:4792: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089392256 (84h52m9.811955296s) [noise=true , address=104.36.82.47:4532 , piece_id=LWHWDYAYV7XR6YW33WDGKUWEOXELRDKVCWF54CXJBU2SEZFCJAXQ, seg_pos={0 0}, upload=81427]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: write tcp 188.166.225.170:38330->104.36.82.47:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:00.089411162 (84h52m9.811974202s) [seg_pos={0 0}, upload=81427]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:28:00.323183015 (84h52m10.045746055s) [upload=81427]: Done uploading segments... Jul 28 00:28:00.323208043 (84h52m10.045771083s) [upload=81427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.235663641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X.X..........X.....X] duration: 30.240088959s FAIL (2025-07-28 00:27:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 81424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: rpc: dial tcp 104.36.82.6:4874: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:00.887031015 (84h51m10.609594045s) [seg_pos={0 0}, upload=81424]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:27:00.887091471 (84h51m10.609654511s) [upload=81424]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:27:00.887096300 (84h51m10.60965934s) [upload=81424]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:27:01.163946963 (84h51m10.886510003s) [upload=81424]: Done uploading segments... Jul 28 00:27:01.163965929 (84h51m10.886528969s) [upload=81424]: Upload finished. err: <nil> Jul 28 00:27:19.597171719 (84h51m29.319734759s) [upload=81431]: Starting upload Jul 28 00:27:19.597272533 (84h51m29.319835573s) [upload=81431]: Done starting upload Jul 28 00:27:19.597365821 (84h51m29.319928861s) [upload=81431]: Uploading segments... Jul 28 00:27:19.597700883 (84h51m29.320263923s) [upload=81431]: Got next segment. Inline: false Jul 28 00:27:19.597736120 (84h51m29.32029915s) [upload=81431]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:27:19.597741259 (84h51m29.320304299s) [upload=81431]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:27:20.045867384 (84h51m29.768430424s) [seg_pos={0 0}, upload=81431]: Begin upload segment... Jul 28 00:27:20.045927990 (84h51m29.76849103s) [seg_pos={0 0}, upload=81431]: Done begin upload segment. Jul 28 00:27:20.045958679 (84h51m29.768521729s) [noise=true , address=77.90.31.15:4698 , piece_id=X5L3W67HMYGRJVZYU72WUJ2ZT3VZ23DRBQJCNUJUV6YZCOL5ICTQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.045965942 (84h51m29.768528982s) [noise=true , address=77.90.31.13:4627 , piece_id=7NR6V4GBJR5WOHQRPDO4VIWT5BPEHWJMRE36UHKIIWSI6VQE6TAA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046055715 (84h51m29.768618755s) [noise=true , address=45.91.116.8:4047 , piece_id=GEDNNAD5CKPNTE7I2X2IK3SCI2G2F3JPKEVXW3HLIO44XOFKIURA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046336261 (84h51m29.768899302s) [noise=true , address=77.90.31.10:4519 , piece_id=N5HFTGGOKYCT42TILLRRHTJRYH4ZRWDQ32ADKNPGJN3ND2NQSUMQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046371579 (84h51m29.768934619s) [noise=true , address=45.91.116.9:4262 , piece_id=6GCZ7XGI4ZRYXBSN3MKUXSOKDFBBMKWWOXDU4VVU56ZDILYUHDEA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046630915 (84h51m29.769193955s) [noise=true , address=45.91.116.7:4010 , piece_id=FQ35UKX2QBOJ7KNNL44QQPNJWYOPBVUKNC67BTPFVQGJ2YUM2CSA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046655041 (84h51m29.769218081s) [noise=true , address=104.36.82.2:4731 , piece_id=KO3XYSDCEOM2YFS3LJOF27UIUULA4ZVRUXFWGPJYR5RJR3BCK45Q, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046905961 (84h51m29.769469001s) [noise=true , address=45.91.116.11:4118 , piece_id=5OBIONSVURIRITMD2HAVHLZTIDIBQOZJENJPKAI5JVKX6NSVG4ZQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.046926390 (84h51m29.76948943s) [noise=true , address=77.90.31.16:4734 , piece_id=LDBACEVIC452LRFQ4NZWTWVJDIQZWD6HJL4XCDS55CJ362LACQBA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047221144 (84h51m29.769784184s) [noise=true , address=104.36.82.10:4077 , piece_id=RTY63EEKUWRHDL6S5WUDY5HAGUFEGFR5BWZO2QAKCBD7UC2JCQRQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047224911 (84h51m29.769787951s) [noise=true , address=104.36.82.4:4802 , piece_id=FFNXZCMDTS632HPHFDWEOXRR3XIMFDG2UEYOAXFFESCQJIWBRWUA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047435665 (84h51m29.769998705s) [noise=true , address=77.90.31.19:4842 , piece_id=SOQDD53C55RO67Q2C5HAYFTSKZP7ZMEWU42OON3YW24MQUMFNJ3A, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047460853 (84h51m29.770023893s) [noise=true , address=77.90.31.11:4555 , piece_id=SZQIQFGZUB225EUHDDT4JMJYMRIJ6FE25BXHM4ZGPDJM4SGTBPQQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047706483 (84h51m29.770269533s) [noise=true , address=77.90.31.21:4879 , piece_id=QBLWJK4D3WXQ4CO7U2XPX262ZDFPLLQKWUUUD43O5BHBCSBZBJLA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.047961571 (84h51m29.770524611s) [noise=true , address=77.90.31.14:4662 , piece_id=PUBSU5WGV2VSPXER4D255XNPMZPBTLOLA5JS5ZBJFJWKWA6CYFBA, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:20.048175260 (84h51m29.7707383s) [noise=true , address=104.36.82.6:4876 , piece_id=3HCF5CQHV7P73Z5I5NVFRUVGPBT5SCU24QXZGOZHRYG3MSA5HOUQ, seg_pos={0 0}, upload=81431]: Uploading piece... Jul 28 00:27:21.770812904 (84h51m31.493375944s) [upload=81431]: Next returned nil segment Jul 28 00:27:21.770831569 (84h51m31.493394609s) [upload=81431]: Waiting for error group managing segments... Jul 28 00:27:22.047721888 (84h51m31.770284928s) [noise=true , address=45.91.116.9:4262 , piece_id=6GCZ7XGI4ZRYXBSN3MKUXSOKDFBBMKWWOXDU4VVU56ZDILYUHDEA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.069919327 (84h51m31.792482367s) [noise=true , address=77.90.31.14:4662 , piece_id=PUBSU5WGV2VSPXER4D255XNPMZPBTLOLA5JS5ZBJFJWKWA6CYFBA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.073840198 (84h51m31.796403248s) [noise=true , address=77.90.31.19:4842 , piece_id=SOQDD53C55RO67Q2C5HAYFTSKZP7ZMEWU42OON3YW24MQUMFNJ3A, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.074205587 (84h51m31.796768637s) [noise=true , address=45.91.116.11:4118 , piece_id=5OBIONSVURIRITMD2HAVHLZTIDIBQOZJENJPKAI5JVKX6NSVG4ZQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.142825163 (84h51m31.865388203s) [noise=true , address=45.91.116.8:4047 , piece_id=GEDNNAD5CKPNTE7I2X2IK3SCI2G2F3JPKEVXW3HLIO44XOFKIURA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.144458218 (84h51m31.867021258s) [noise=true , address=77.90.31.16:4734 , piece_id=LDBACEVIC452LRFQ4NZWTWVJDIQZWD6HJL4XCDS55CJ362LACQBA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.172639546 (84h51m31.895202586s) [noise=true , address=77.90.31.10:4519 , piece_id=N5HFTGGOKYCT42TILLRRHTJRYH4ZRWDQ32ADKNPGJN3ND2NQSUMQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.218333209 (84h51m31.940896249s) [noise=true , address=77.90.31.13:4627 , piece_id=7NR6V4GBJR5WOHQRPDO4VIWT5BPEHWJMRE36UHKIIWSI6VQE6TAA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.223604784 (84h51m31.946167824s) [noise=true , address=77.90.31.21:4879 , piece_id=QBLWJK4D3WXQ4CO7U2XPX262ZDFPLLQKWUUUD43O5BHBCSBZBJLA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.236036920 (84h51m31.958599961s) [noise=true , address=77.90.31.15:4698 , piece_id=X5L3W67HMYGRJVZYU72WUJ2ZT3VZ23DRBQJCNUJUV6YZCOL5ICTQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.251041218 (84h51m31.973604258s) [noise=true , address=45.91.116.7:4010 , piece_id=FQ35UKX2QBOJ7KNNL44QQPNJWYOPBVUKNC67BTPFVQGJ2YUM2CSA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:22.279611289 (84h51m32.002174329s) [noise=true , address=77.90.31.11:4555 , piece_id=SZQIQFGZUB225EUHDDT4JMJYMRIJ6FE25BXHM4ZGPDJM4SGTBPQQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: <nil> Jul 28 00:27:48.947845071 (84h51m58.670408111s) [noise=true , address=104.36.82.10:4077 , piece_id=RTY63EEKUWRHDL6S5WUDY5HAGUFEGFR5BWZO2QAKCBD7UC2JCQRQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:48.947912730 (84h51m58.67047577s) [noise=true , address=104.36.82.4:4802 , piece_id=FFNXZCMDTS632HPHFDWEOXRR3XIMFDG2UEYOAXFFESCQJIWBRWUA, seg_pos={0 0}, upload=81431]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:57482->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:48.947943999 (84h51m58.670507039s) [noise=true , address=104.36.82.6:4876 , piece_id=3HCF5CQHV7P73Z5I5NVFRUVGPBT5SCU24QXZGOZHRYG3MSA5HOUQ, seg_pos={0 0}, upload=81431]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:51704->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:48.948013092 (84h51m58.670576132s) [noise=true , address=104.36.82.2:4731 , piece_id=KO3XYSDCEOM2YFS3LJOF27UIUULA4ZVRUXFWGPJYR5RJR3BCK45Q, seg_pos={0 0}, upload=81431]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:36822->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:48.948060071 (84h51m58.670623121s) [seg_pos={0 0}, upload=81431]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:27:49.184974758 (84h51m58.907537808s) [upload=81431]: Done uploading segments... Jul 28 00:27:49.185004114 (84h51m58.907567154s) [upload=81431]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.240106604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X............X......................] duration: 1.863107069s FAIL (2025-07-28 00:27:28 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.863130713s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................X.......X.XX....X...] duration: 5.819179534s FAIL (2025-07-28 00:27:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.819205284s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.X..........X.....] duration: 5.841729997s FAIL (2025-07-28 00:27:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== M3CTHVHBOICECUDARKRK2KDCQAJ4GZUI543EMR72LUNYHSU62IWA, seg_pos={0 0}, upload=81361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Jul 28 00:22:41.678787344 (84h46m51.401350384s) [seg_pos={0 0}, upload=81361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:22:41.678874520 (84h46m51.40143756s) [upload=81361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:22:41.678879389 (84h46m51.401442429s) [upload=81361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:22:41.957215845 (84h46m51.679778875s) [upload=81361]: Done uploading segments... Jul 28 00:22:41.957233678 (84h46m51.679796718s) [upload=81361]: Upload finished. err: <nil> Jul 28 00:23:37.076289486 (84h47m46.798852536s) [upload=81374]: Starting upload Jul 28 00:23:37.076409375 (84h47m46.798972425s) [upload=81374]: Done starting upload Jul 28 00:23:37.076524956 (84h47m46.799087996s) [upload=81374]: Uploading segments... Jul 28 00:23:37.076610781 (84h47m46.799173821s) [upload=81374]: Got next segment. Inline: false Jul 28 00:23:37.076641660 (84h47m46.7992047s) [upload=81374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:23:37.076646429 (84h47m46.799209459s) [upload=81374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:23:37.317328312 (84h47m47.039891352s) [seg_pos={0 0}, upload=81374]: Begin upload segment... Jul 28 00:23:37.317375502 (84h47m47.039938542s) [seg_pos={0 0}, upload=81374]: Done begin upload segment. Jul 28 00:23:37.317387365 (84h47m47.039950405s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317398867 (84h47m47.039961907s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317741793 (84h47m47.040304833s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317771279 (84h47m47.040334329s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318039613 (84h47m47.040602653s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318294580 (84h47m47.04085762s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318532536 (84h47m47.041095566s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318775812 (84h47m47.041338852s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318964052 (84h47m47.041527092s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319152892 (84h47m47.041715932s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319350842 (84h47m47.041913882s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319540073 (84h47m47.042103113s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319723063 (84h47m47.042286094s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319905383 (84h47m47.042468423s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320084705 (84h47m47.042647745s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320267355 (84h47m47.042830395s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:39.064964023 (84h47m48.787527053s) [upload=81374]: Next returned nil segment Jul 28 00:23:39.064981687 (84h47m48.787544727s) [upload=81374]: Waiting for error group managing segments... Jul 28 00:23:39.340867483 (84h47m49.063430523s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.356965262 (84h47m49.079528302s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.362359653 (84h47m49.084922693s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.373141581 (84h47m49.095704651s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.430640531 (84h47m49.153203571s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.445133640 (84h47m49.16769667s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.454045370 (84h47m49.1766084s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.503182252 (84h47m49.225745292s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.519937578 (84h47m49.242500618s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.522381895 (84h47m49.244944935s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.531306760 (84h47m49.2538698s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.602626742 (84h47m49.325189792s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:24:06.434846113 (84h48m16.157409153s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:45336->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434872944 (84h48m16.157435984s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434917430 (84h48m16.15748047s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:60368->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434930544 (84h48m16.157493584s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55716->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434955502 (84h48m16.157518542s) [seg_pos={0 0}, upload=81374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:24:06.663297936 (84h48m16.385860976s) [upload=81374]: Done uploading segments... Jul 28 00:24:06.663325118 (84h48m16.385888158s) [upload=81374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841754403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [......................................X...XXXXXXXXXXXXXXXXXX] duration: 1m0.230038833s FAIL (2025-07-28 00:26:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 6IDGM5H3M4ZH4UAABZXEZA6VJWGGKLXHQ, seg_pos={0 0}, upload=81409]: Uploading piece... Jul 28 00:26:00.706330097 (84h50m10.428893138s) [noise=true , address=104.36.82.5:4859 , piece_id=DDME4MPGJAXZPMJQYM2TUHXH7Y3UACEF7MG5DK4IXHWBHG3SL5WA, seg_pos={0 0}, upload=81409]: Uploading piece... Jul 28 00:26:00.706898315 (84h50m10.429461355s) [noise=true , address=104.36.82.12:4162 , piece_id=GDUOLKYFA5LWCRVYODDX2KOFF5UV2NBPZFYCG7ZQRAHQ4TQB3GVA, seg_pos={0 0}, upload=81409]: Uploading piece... Jul 28 00:26:00.707467844 (84h50m10.430030884s) [noise=true , address=104.36.82.14:4247 , piece_id=SYXBQNMTRMEXJTB3K6YSMY74H7DJD4NMBNS3XA3YVW7OVIMLNF2Q, seg_pos={0 0}, upload=81409]: Uploading piece... Jul 28 00:26:00.708015883 (84h50m10.430578923s) [noise=true , address=104.36.82.49:4986 , piece_id=GU3YZEKVO7ZKZBXYTHPZYEGOWOKOREDUBGN3P6QYJU7AZ6FWKL5A, seg_pos={0 0}, upload=81409]: Uploading piece... Jul 28 00:26:00.708448911 (84h50m10.431011951s) [noise=true , address=104.36.82.34:4576 , piece_id=QYTTWWN3SD4KUCCKOU2T3Z3VPRLTPTYG5525325JHJWN5KLMS77Q, seg_pos={0 0}, upload=81409]: Uploading piece... Jul 28 00:26:02.224088658 (84h50m11.946651698s) [upload=81409]: Next returned nil segment Jul 28 00:26:02.224106062 (84h50m11.946669112s) [upload=81409]: Waiting for error group managing segments... Jul 28 00:26:06.860723527 (84h50m16.583286567s) [noise=true , address=79.127.201.215:4352 , piece_id=WN42ADE2BUJQIZM53ZFJWRY45RHOUNYAXHCIABMTIGIR2HCTWIRA, seg_pos={0 0}, upload=81409]: Done uploading piece. err: <nil> Jul 28 00:26:59.614074079 (84h51m9.33663711s) [noise=true , address=104.36.82.5:4859 , piece_id=DDME4MPGJAXZPMJQYM2TUHXH7Y3UACEF7MG5DK4IXHWBHG3SL5WA, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: write tcp 188.166.225.170:54252->104.36.82.5:4859: use of closed network connection; conn closed: op Jul 28 00:26:59.614108555 (84h51m9.336671595s) [noise=true , address=104.36.82.42:4829 , piece_id=4LW3LNJOZFFUJWVHMPGAWTOYTL65XWHLWHDY52KJUNU5FG3MCYYA, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: write tcp 188.166.225.170:36052->104.36.82.42:4829: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614150315 (84h51m9.336713355s) [noise=true , address=104.36.82.16:4312 , piece_id=5OXGGJQAQ2ZVMMOPFU4HYXGLVUHW5OAVTKYHHVGOQVQIHRTGLSVQ, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:104.36.82.16:4312): protocol: rpc: dial tcp 104.36.82.16:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614183870 (84h51m9.33674691s) [noise=true , address=104.36.82.10:4096 , piece_id=PIZQUZ3LWXFOBSHYCWAMWGJSRDKWWVBX6MTH3BV4CG5ZWV6XXI6A, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: write tcp 188.166.225.170:49032->104.36.82.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614225719 (84h51m9.336788759s) [noise=true , address=104.36.82.41:4787 , piece_id=53MATBJSS2GWMC2OR54VYK4D7MB2TN5X2HVGC7KJQV65K5MIVBWQ, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:12aKqg1aNsv8Yi7fE4asGz8bijcPewWDvUukKcf5SFnzgARju43, address:104.36.82.41:4787): protocol: write tcp 188.166.225.170:59830->104.36.82.41:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614269513 (84h51m9.336832553s) [noise=true , address=104.36.82.43:4864 , piece_id=MTUZ66DXKV2UUWGDPKKFAFSVACMDBFUMVEXYWUMV6L7S5GOODCQA, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC, address:104.36.82.43:4864): protocol: write tcp 188.166.225.170:34398->104.36.82.43:4864: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614308538 (84h51m9.336871568s) [noise=true , address=104.36.82.12:4162 , piece_id=GDUOLKYFA5LWCRVYODDX2KOFF5UV2NBPZFYCG7ZQRAHQ4TQB3GVA, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:104.36.82.12:4162): protocol: write tcp 188.166.225.170:43456->104.36.82.12:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614337703 (84h51m9.336900743s) [noise=true , address=104.36.82.11:4133 , piece_id=KIIJT5DHP72KNEJSDDOBND7CUTC4SWL2HIU53ZKEIXSSUJPDWFTQ, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: write tcp 188.166.225.170:45208->104.36.82.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614342783 (84h51m9.336905823s) [noise=true , address=104.36.82.49:4986 , piece_id=GU3YZEKVO7ZKZBXYTHPZYEGOWOKOREDUBGN3P6QYJU7AZ6FWKL5A, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:44998->104.36.82.49:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614366118 (84h51m9.336929158s) [noise=true , address=104.36.82.53:4132 , piece_id=WYQZZ4NZRXMSAMGYHMJ4UZLFSDNFFDMT6G7UZWFHECHPPB5SZ3CQ, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: write tcp 188.166.225.170:35570->104.36.82.53:4132: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614371939 (84h51m9.336934969s) [noise=true , address=104.36.82.55:4224 , piece_id=A6MIPLGTDZFT4SCXKWVFBSIHCNQODGVYI2YRPHEN3VENVDYDOWOQ, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 188.166.225.170:37706->104.36.82.55:4224: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614399371 (84h51m9.336962411s) [noise=true , address=104.36.82.56:4230 , piece_id=HNOSBGUWFMKDPBRN2MPJKAQHOKYOY2FFEOIGWUGLO42C4FFB3E4Q, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: write tcp 188.166.225.170:57924->104.36.82.56:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614407157 (84h51m9.336970187s) [noise=true , address=104.36.82.54:4190 , piece_id=IORS7JMGNIXSNTMNJQC2WVVL6SKI3N4BRSBG47PSGXBQPS5D3V5A, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: write tcp 188.166.225.170:58904->104.36.82.54:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614426273 (84h51m9.336989313s) [noise=true , address=104.36.82.52:4096 , piece_id=JAVFYESBBJADSYCFGFP6IDGM5H3M4ZH4UAABZXEZA6VJWGGKLXHQ, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: write tcp 188.166.225.170:42492->104.36.82.52:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614450379 (84h51m9.337013419s) [noise=true , address=104.36.82.14:4247 , piece_id=SYXBQNMTRMEXJTB3K6YSMY74H7DJD4NMBNS3XA3YVW7OVIMLNF2Q, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:104.36.82.14:4247): protocol: write tcp 188.166.225.170:57354->104.36.82.14:4247: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614474575 (84h51m9.337037615s) [noise=true , address=104.36.82.34:4576 , piece_id=QYTTWWN3SD4KUCCKOU2T3Z3VPRLTPTYG5525325JHJWN5KLMS77Q, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:122bMTJLxntv7VEY56zSPLZm5wZHAH3rnE8UZHNCjRhd7P9oKus, address:104.36.82.34:4576): protocol: write tcp 188.166.225.170:33806->104.36.82.34:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614479154 (84h51m9.337042184s) [noise=true , address=104.36.82.44:4885 , piece_id=KWGJLFUTQI75CYBETYRUFP6SFG6N7WZW6R7Z3GNZR3OH23KG5DNQ, seg_pos={0 0}, upload=81409]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: write tcp 188.166.225.170:44504->104.36.82.44:4885: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:59.614491979 (84h51m9.337055019s) [seg_pos={0 0}, upload=81409]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:26:59.842727208 (84h51m9.565290248s) [upload=81409]: Done uploading segments... Jul 28 00:26:59.842776653 (84h51m9.565339683s) [upload=81409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.230061197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................X............X.....................] duration: 2.002911266s FAIL (2025-07-28 00:26:35 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.002936213s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................X.......X.XX....X..] duration: 5.920593632s FAIL (2025-07-28 00:26:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.920623729s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X.......X.XX....X.] duration: 5.953898803s FAIL (2025-07-28 00:26:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.9539232s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.X..........X....] duration: 5.992416857s FAIL (2025-07-28 00:26:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== M3CTHVHBOICECUDARKRK2KDCQAJ4GZUI543EMR72LUNYHSU62IWA, seg_pos={0 0}, upload=81361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Jul 28 00:22:41.678787344 (84h46m51.401350384s) [seg_pos={0 0}, upload=81361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:22:41.678874520 (84h46m51.40143756s) [upload=81361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:22:41.678879389 (84h46m51.401442429s) [upload=81361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:22:41.957215845 (84h46m51.679778875s) [upload=81361]: Done uploading segments... Jul 28 00:22:41.957233678 (84h46m51.679796718s) [upload=81361]: Upload finished. err: <nil> Jul 28 00:23:37.076289486 (84h47m46.798852536s) [upload=81374]: Starting upload Jul 28 00:23:37.076409375 (84h47m46.798972425s) [upload=81374]: Done starting upload Jul 28 00:23:37.076524956 (84h47m46.799087996s) [upload=81374]: Uploading segments... Jul 28 00:23:37.076610781 (84h47m46.799173821s) [upload=81374]: Got next segment. Inline: false Jul 28 00:23:37.076641660 (84h47m46.7992047s) [upload=81374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:23:37.076646429 (84h47m46.799209459s) [upload=81374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:23:37.317328312 (84h47m47.039891352s) [seg_pos={0 0}, upload=81374]: Begin upload segment... Jul 28 00:23:37.317375502 (84h47m47.039938542s) [seg_pos={0 0}, upload=81374]: Done begin upload segment. Jul 28 00:23:37.317387365 (84h47m47.039950405s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317398867 (84h47m47.039961907s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317741793 (84h47m47.040304833s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317771279 (84h47m47.040334329s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318039613 (84h47m47.040602653s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318294580 (84h47m47.04085762s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318532536 (84h47m47.041095566s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318775812 (84h47m47.041338852s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318964052 (84h47m47.041527092s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319152892 (84h47m47.041715932s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319350842 (84h47m47.041913882s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319540073 (84h47m47.042103113s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319723063 (84h47m47.042286094s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319905383 (84h47m47.042468423s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320084705 (84h47m47.042647745s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320267355 (84h47m47.042830395s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:39.064964023 (84h47m48.787527053s) [upload=81374]: Next returned nil segment Jul 28 00:23:39.064981687 (84h47m48.787544727s) [upload=81374]: Waiting for error group managing segments... Jul 28 00:23:39.340867483 (84h47m49.063430523s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.356965262 (84h47m49.079528302s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.362359653 (84h47m49.084922693s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.373141581 (84h47m49.095704651s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.430640531 (84h47m49.153203571s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.445133640 (84h47m49.16769667s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.454045370 (84h47m49.1766084s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.503182252 (84h47m49.225745292s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.519937578 (84h47m49.242500618s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.522381895 (84h47m49.244944935s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.531306760 (84h47m49.2538698s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.602626742 (84h47m49.325189792s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:24:06.434846113 (84h48m16.157409153s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:45336->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434872944 (84h48m16.157435984s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434917430 (84h48m16.15748047s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:60368->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434930544 (84h48m16.157493584s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55716->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434955502 (84h48m16.157518542s) [seg_pos={0 0}, upload=81374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:24:06.663297936 (84h48m16.385860976s) [upload=81374]: Done uploading segments... Jul 28 00:24:06.663325118 (84h48m16.385888158s) [upload=81374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.992444391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................X............X....................] duration: 1.939475266s FAIL (2025-07-28 00:26:07 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.93950965s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......................................X...XXXXXXXXXXXXXXXXX] duration: 1m0.236474257s FAIL (2025-07-28 00:25:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 1 , piece_id=VTDXU26RJTFYH7AMTBQZWQ3QKYVDZFNHPOCJHFRDMH5AQ2FULMRA, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.220321220 (84h49m9.94288427s) [noise=true , address=104.36.82.50:4023 , piece_id=QPF4KEWUDQJTIQRTDT7S5SWXHFOYGKKQURSMN4WBCX24M6GJQ6TA, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.222382443 (84h49m9.944945483s) [noise=true , address=104.36.82.15:4269 , piece_id=CDRX7KNIW75FVJWVXLGMM2A2MYOIA36I3QSAXEMOOMOD6KWRNYVQ, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.223240054 (84h49m9.945803094s) [noise=true , address=79.127.201.215:4353 , piece_id=3S6Q5RBJ7CUNW6DOJ4RDYWMFBKV4T5MYSZTWTCUHWJR7IML4TSRA, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.223517074 (84h49m9.946080114s) [noise=true , address=104.36.82.16:4316 , piece_id=J6BSARM5TOFVMLF6HI2DCZLX3EM4XXOJ4DS2OMIOSMVQW6HRXTOA, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.223786409 (84h49m9.946349449s) [noise=true , address=104.36.82.54:4172 , piece_id=34MW52WES675ULL7RPI27KJFS4IYXG7QSM4OTGWU7SYPDDIBGVXQ, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.224019285 (84h49m9.946582325s) [noise=true , address=64.49.13.138:4482 , piece_id=ETGNED6LEFUADOW2QLXHNG2SUDC4ZBD5HLS5Y5EJTRMQXVM3IV4A, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.224236911 (84h49m9.946799951s) [noise=true , address=79.127.205.235:4669 , piece_id=6AFZVGHOP6KZJPECD2UIW33B6F6Q26AA5ZIC4BW7UWO2J7FZ66WQ, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:01.277058093 (84h49m10.999621133s) [noise=true , address=64.49.13.138:4482 , piece_id=ETGNED6LEFUADOW2QLXHNG2SUDC4ZBD5HLS5Y5EJTRMQXVM3IV4A, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 28 00:25:01.349103484 (84h49m11.071666524s) [upload=81398]: Next returned nil segment Jul 28 00:25:01.349124555 (84h49m11.071687625s) [upload=81398]: Waiting for error group managing segments... Jul 28 00:25:01.664316903 (84h49m11.386879953s) [noise=true , address=79.127.205.235:4669 , piece_id=6AFZVGHOP6KZJPECD2UIW33B6F6Q26AA5ZIC4BW7UWO2J7FZ66WQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: <nil> Jul 28 00:25:03.593833469 (84h49m13.316396519s) [noise=true , address=79.127.201.215:4353 , piece_id=3S6Q5RBJ7CUNW6DOJ4RDYWMFBKV4T5MYSZTWTCUHWJR7IML4TSRA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: <nil> Jul 28 00:25:59.128138643 (84h50m8.850701683s) [noise=true , address=104.36.82.55:4193 , piece_id=C2AD5AHRRCVIOB6NF3T2JBDULPMDFGIT3WKYNLVSUZDEDPZY4CUQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: write tcp 188.166.225.170:54014->104.36.82.55:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128195162 (84h50m8.850758202s) [noise=true , address=104.36.82.53:4133 , piece_id=UAPKBBUFJEA2ZZ423UPHCXVTQRFGHUG3GGYKULYYN4OIY4XJC4VA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: write tcp 188.166.225.170:51602->104.36.82.53:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128230790 (84h50m8.85079383s) [noise=true , address=104.36.82.47:4524 , piece_id=BPLN3IAWLKWDARSLQI6KV55VZCONMK466LDBNPBZ6WBJV6FZGGRQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 188.166.225.170:48814->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128281688 (84h50m8.850844718s) [noise=true , address=104.36.82.16:4316 , piece_id=J6BSARM5TOFVMLF6HI2DCZLX3EM4XXOJ4DS2OMIOSMVQW6HRXTOA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12PPnXDzwXDVDKemw483GXuzQrPN48JJrjeeD7az2NPNawHNaXf, address:104.36.82.16:4316): protocol: rpc: dial tcp 104.36.82.16:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128318889 (84h50m8.850881929s) [noise=true , address=104.36.82.14:4231 , piece_id=FO6CDNXEKQ6H6QZYRY2FIWR7DVHKX56CSUAMELSDL5ZGJXLGKH3A, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1ixY5pokE6MtrVH9FpKc1tGZhEAaW5brbzBeeGmYUczK3z7tCQ, address:104.36.82.14:4231): protocol: write tcp 188.166.225.170:40198->104.36.82.14:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128397740 (84h50m8.85096078s) [noise=true , address=104.36.82.4:4833 , piece_id=3MVB6JNGMHQPR4W4C4OPNBLYG66YOXCI6ZOT5AB5O3OEOSI5CQOA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: write tcp 188.166.225.170:54030->104.36.82.4:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128450030 (84h50m8.85101307s) [noise=true , address=104.36.82.35:4589 , piece_id=POFSWYNKJMSV6W3CWKW32XCCI5JCEYCU3TJKYWUEREK4CJZRTHVQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: write tcp 188.166.225.170:51890->104.36.82.35:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128503592 (84h50m8.851066632s) [noise=true , address=104.36.82.52:4107 , piece_id=6MRSKLF5ZMDEXJU4PJPP5MT5KHG7AZXTQYEHJ4STYUAZOHOXVSKQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: write tcp 188.166.225.170:40086->104.36.82.52:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128556944 (84h50m8.851119995s) [noise=true , address=104.36.82.12:4171 , piece_id=VTDXU26RJTFYH7AMTBQZWQ3QKYVDZFNHPOCJHFRDMH5AQ2FULMRA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1z2Fh5fPFTb9VZX6to5hdUYUGst63Ti1BVtdVtYk7vkrUqXMKM, address:104.36.82.12:4171): protocol: write tcp 188.166.225.170:45684->104.36.82.12:4171: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128600056 (84h50m8.851163096s) [noise=true , address=104.36.82.54:4172 , piece_id=34MW52WES675ULL7RPI27KJFS4IYXG7QSM4OTGWU7SYPDDIBGVXQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: rpc: dial tcp 104.36.82.54:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128632218 (84h50m8.851195248s) [noise=true , address=104.36.82.50:4023 , piece_id=QPF4KEWUDQJTIQRTDT7S5SWXHFOYGKKQURSMN4WBCX24M6GJQ6TA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12c8EYHav34x4uWw8LnwFZqMdDTfoJDzJS3K6rA8dAQv6nHrK4v, address:104.36.82.50:4023): protocol: rpc: dial tcp 104.36.82.50:4023: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128636537 (84h50m8.851199577s) [noise=true , address=104.36.82.13:4218 , piece_id=R4XMR6N3SENNU7EXLWVIZ7TLLEELZJ3CEMDBYN7PTAC4SOH7AS2A, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc, address:104.36.82.13:4218): protocol: write tcp 188.166.225.170:56396->104.36.82.13:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128676383 (84h50m8.851239423s) [noise=true , address=104.36.82.46:4489 , piece_id=VLIPR6A2NXKAEEWSXSGOEXPTFAOHFOTE2EAYJQN4CBBCMV2Z74DA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: write tcp 188.166.225.170:49202->104.36.82.46:4489: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128683456 (84h50m8.851246496s) [noise=true , address=104.36.82.45:4906 , piece_id=Z5VX4AFLM72RVM26K5KOC6NM7IGHTTR2ZLWAF6YUIPW6JRAEOD5A, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:104.36.82.45:4906): protocol: write tcp 188.166.225.170:46586->104.36.82.45:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128705598 (84h50m8.851268639s) [noise=true , address=104.36.82.15:4269 , piece_id=CDRX7KNIW75FVJWVXLGMM2A2MYOIA36I3QSAXEMOOMOD6KWRNYVQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12V8v5nzyyouSqndATi4fRJUC6HgFbhguBjnmwpki48zdRBTcgC, address:104.36.82.15:4269): protocol: rpc: dial tcp 104.36.82.15:4269: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128785031 (84h50m8.851348071s) [seg_pos={0 0}, upload=81398]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:25:59.362015718 (84h50m9.084578758s) [upload=81398]: Done uploading segments... Jul 28 00:25:59.362039173 (84h50m9.084602213s) [upload=81398]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.236488885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.X..........X...] duration: 5.767887802s FAIL (2025-07-28 00:25:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== M3CTHVHBOICECUDARKRK2KDCQAJ4GZUI543EMR72LUNYHSU62IWA, seg_pos={0 0}, upload=81361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Jul 28 00:22:41.678787344 (84h46m51.401350384s) [seg_pos={0 0}, upload=81361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:22:41.678874520 (84h46m51.40143756s) [upload=81361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:22:41.678879389 (84h46m51.401442429s) [upload=81361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:22:41.957215845 (84h46m51.679778875s) [upload=81361]: Done uploading segments... Jul 28 00:22:41.957233678 (84h46m51.679796718s) [upload=81361]: Upload finished. err: <nil> Jul 28 00:23:37.076289486 (84h47m46.798852536s) [upload=81374]: Starting upload Jul 28 00:23:37.076409375 (84h47m46.798972425s) [upload=81374]: Done starting upload Jul 28 00:23:37.076524956 (84h47m46.799087996s) [upload=81374]: Uploading segments... Jul 28 00:23:37.076610781 (84h47m46.799173821s) [upload=81374]: Got next segment. Inline: false Jul 28 00:23:37.076641660 (84h47m46.7992047s) [upload=81374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:23:37.076646429 (84h47m46.799209459s) [upload=81374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:23:37.317328312 (84h47m47.039891352s) [seg_pos={0 0}, upload=81374]: Begin upload segment... Jul 28 00:23:37.317375502 (84h47m47.039938542s) [seg_pos={0 0}, upload=81374]: Done begin upload segment. Jul 28 00:23:37.317387365 (84h47m47.039950405s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317398867 (84h47m47.039961907s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317741793 (84h47m47.040304833s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317771279 (84h47m47.040334329s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318039613 (84h47m47.040602653s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318294580 (84h47m47.04085762s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318532536 (84h47m47.041095566s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318775812 (84h47m47.041338852s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318964052 (84h47m47.041527092s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319152892 (84h47m47.041715932s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319350842 (84h47m47.041913882s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319540073 (84h47m47.042103113s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319723063 (84h47m47.042286094s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319905383 (84h47m47.042468423s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320084705 (84h47m47.042647745s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320267355 (84h47m47.042830395s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:39.064964023 (84h47m48.787527053s) [upload=81374]: Next returned nil segment Jul 28 00:23:39.064981687 (84h47m48.787544727s) [upload=81374]: Waiting for error group managing segments... Jul 28 00:23:39.340867483 (84h47m49.063430523s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.356965262 (84h47m49.079528302s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.362359653 (84h47m49.084922693s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.373141581 (84h47m49.095704651s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.430640531 (84h47m49.153203571s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.445133640 (84h47m49.16769667s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.454045370 (84h47m49.1766084s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.503182252 (84h47m49.225745292s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.519937578 (84h47m49.242500618s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.522381895 (84h47m49.244944935s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.531306760 (84h47m49.2538698s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.602626742 (84h47m49.325189792s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:24:06.434846113 (84h48m16.157409153s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:45336->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434872944 (84h48m16.157435984s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434917430 (84h48m16.15748047s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:60368->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434930544 (84h48m16.157493584s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55716->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434955502 (84h48m16.157518542s) [seg_pos={0 0}, upload=81374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:24:06.663297936 (84h48m16.385860976s) [upload=81374]: Done uploading segments... Jul 28 00:24:06.663325118 (84h48m16.385888158s) [upload=81374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767917257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................X.......X.XX....X] duration: 30.247971673s FAIL (2025-07-28 00:25:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.247990067s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)