Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............X.X......XX.X...................................] duration: 5.839155991s FAIL (2025-07-18 02:48:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.839183604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XX.......XXXX.................................] duration: 5.917583593s FAIL (2025-07-18 02:48:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.917623591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............X.X......XX.X..................................] duration: 5.805071958s FAIL (2025-07-18 02:47:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.805097386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XX.......XXXX................................] duration: 6.115331537s FAIL (2025-07-18 02:47:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.115361766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X.X......XX.X.................................] duration: 5.928066142s FAIL (2025-07-18 02:47:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.928100808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............X.X......XX.X................................] duration: 5.790088869s FAIL (2025-07-18 02:46:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.790117544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................XX.......XXXX...............................] duration: 5.864354292s FAIL (2025-07-18 02:46:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.864381543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................X.X......XX.X...............................] duration: 5.757690369s FAIL (2025-07-18 02:45:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.75771737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................XX.......XXXX..............................] duration: 5.854349068s FAIL (2025-07-18 02:45:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854378113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................X.X......XX.X..............................] duration: 5.838188631s FAIL (2025-07-18 02:45:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.838222697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................X.X......XX.X.............................] duration: 5.706906495s FAIL (2025-07-18 02:44:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.706949257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................XX.......XXXX.............................] duration: 5.874769879s FAIL (2025-07-18 02:44:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.874806219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................X.X......XX.X............................] duration: 5.896230502s FAIL (2025-07-18 02:44:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.896256351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XX.......XXXX............................] duration: 6.102194429s FAIL (2025-07-18 02:43:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.102221681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................X.X......XX.X...........................] duration: 5.826297005s FAIL (2025-07-18 02:42:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.826321793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XX.......XXXX...........................] duration: 6.056016447s FAIL (2025-07-18 02:42:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.056040584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................X.X......XX.X..........................] duration: 5.684544408s FAIL (2025-07-18 02:42:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.684573002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XX.......XXXX..........................] duration: 5.875996711s FAIL (2025-07-18 02:41:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.876018542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................X.X......XX.X.........................] duration: 5.835635642s FAIL (2025-07-18 02:41:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.835684636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XX.......XXXX.........................] duration: 5.944006312s FAIL (2025-07-18 02:40:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.944035016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................X.X......XX.X........................] duration: 5.689224932s FAIL (2025-07-18 02:40:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.689254059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................X.X......XX.X.......................] duration: 5.760848521s FAIL (2025-07-18 02:40:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.760878588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XX.......XXXX........................] duration: 5.929862322s FAIL (2025-07-18 02:39:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.929889554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........................X.X......XX.X......................] duration: 5.81603129s FAIL (2025-07-18 02:39:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.816055787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XX.......XXXX.......................] duration: 5.915713058s FAIL (2025-07-18 02:39:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.915737254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................X.X......XX.X.....................] duration: 5.762517376s FAIL (2025-07-18 02:38:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.762581428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........................X.X......XX.X....................] duration: 5.733399744s FAIL (2025-07-18 02:38:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.733427869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XX.......XXXX......................] duration: 5.92882027s FAIL (2025-07-18 02:38:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.92884617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XX.......XXXX.....................] duration: 5.964979846s FAIL (2025-07-18 02:37:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.965007839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............................X.X......XX.X...................] duration: 5.668723066s FAIL (2025-07-18 02:37:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.668750418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................XX.......XXXX....................] duration: 5.860599298s FAIL (2025-07-18 02:36:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.860650246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XX.......XXXX...................] duration: 5.949454139s FAIL (2025-07-18 02:36:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.949478425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............................X.X......XX.X..................] duration: 5.896810409s FAIL (2025-07-18 02:36:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.896838833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XX.......XXXX..................] duration: 6.020414488s FAIL (2025-07-18 02:35:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.020442702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............................X.X......XX.X.................] duration: 5.814302203s FAIL (2025-07-18 02:35:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.814327801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XX.......XXXX.................] duration: 5.926036527s FAIL (2025-07-18 02:35:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.926067476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XX.......XXXX................] duration: 5.965245686s FAIL (2025-07-18 02:34:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.965274332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............................X.X......XX.X................] duration: 5.787638018s FAIL (2025-07-18 02:34:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.787676061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XX.......XXXX...............] duration: 5.928794472s FAIL (2025-07-18 02:33:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.928829087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................................X.X......XX.X...............] duration: 5.908532505s FAIL (2025-07-18 02:33:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.908558283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XX.......XXXX..............] duration: 5.832415665s FAIL (2025-07-18 02:32:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832442165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................................X.X......XX.X..............] duration: 5.795144598s FAIL (2025-07-18 02:32:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.795172522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................................X.X......XX.X.............] duration: 6.039112397s FAIL (2025-07-18 02:31:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.039138487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................XX.......XXXX.............] duration: 5.999910281s FAIL (2025-07-18 02:31:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.999949786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................XX.......XXXX............] duration: 5.859212241s FAIL (2025-07-18 02:31:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.859239682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................................X.X......XX.X............] duration: 5.701419861s FAIL (2025-07-18 02:30:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.701443896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................XX.......XXXX...........] duration: 5.855478425s FAIL (2025-07-18 02:30:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.855507371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................................X.X......XX.X...........] duration: 5.915333508s FAIL (2025-07-18 02:30:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.915363035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X.X......XX.X..........] duration: 5.722984354s FAIL (2025-07-18 02:29:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.723005364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XX.......XXXX..........] duration: 6.004628928s FAIL (2025-07-18 02:29:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.004655109s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)