Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X......XX.X................................................] duration: 5.697404773s FAIL (2025-07-18 02:58:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.697431604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X.X......XX.X...............................................] duration: 5.689681347s FAIL (2025-07-18 02:58:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.689707018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.......XXXX.............................................] duration: 5.861304552s FAIL (2025-07-18 02:58:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.861333697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X.X......XX.X..............................................] duration: 5.755759713s FAIL (2025-07-18 02:57:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.75578972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX.......XXXX............................................] duration: 5.869380405s FAIL (2025-07-18 02:57:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.869424219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X.X......XX.X.............................................] duration: 5.717575053s FAIL (2025-07-18 02:57:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.717616854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.......XXXX...........................................] duration: 5.84823588s FAIL (2025-07-18 02:57:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.848265126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X.X......XX.X............................................] duration: 5.710424082s FAIL (2025-07-18 02:57:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.710459119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X.X......XX.X...........................................] duration: 5.748711133s FAIL (2025-07-18 02:56:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.748740869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX.......XXXX..........................................] duration: 5.88054478s FAIL (2025-07-18 02:56:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.880574136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X.X......XX.X..........................................] duration: 5.862127846s FAIL (2025-07-18 02:55:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.86219347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX.......XXXX.........................................] duration: 5.898648015s FAIL (2025-07-18 02:55:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.898676068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X.X......XX.X.........................................] duration: 5.719709047s FAIL (2025-07-18 02:54:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.719736909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XX.......XXXX........................................] duration: 5.909949379s FAIL (2025-07-18 02:54:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.909978364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X.X......XX.X........................................] duration: 5.642368028s FAIL (2025-07-18 02:53:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.642399037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XX.......XXXX.......................................] duration: 5.884073584s FAIL (2025-07-18 02:53:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.884101267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XX.......XXXX......................................] duration: 5.860315477s FAIL (2025-07-18 02:52:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.860372576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........X.X......XX.X.......................................] duration: 5.65671139s FAIL (2025-07-18 02:52:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.65673695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XX.......XXXX.....................................] duration: 5.916217683s FAIL (2025-07-18 02:51:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.916243481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........X.X......XX.X......................................] duration: 5.778740108s FAIL (2025-07-18 02:51:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.778768532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XX.......XXXX....................................] duration: 5.938101686s FAIL (2025-07-18 02:51:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.938134377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........X.X......XX.X.....................................] duration: 5.947479249s FAIL (2025-07-18 02:50:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.947512994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XX.......XXXX...................................] duration: 5.874650342s FAIL (2025-07-18 02:49:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.874680259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........X.X......XX.X....................................] duration: 5.89026299s FAIL (2025-07-18 02:49:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.890288639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XX.......XXXX..................................] duration: 5.893591065s FAIL (2025-07-18 02:49:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.893618035s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............X.X......XX.X...................................] duration: 5.839155991s FAIL (2025-07-18 02:48:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.839183604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XX.......XXXX.................................] duration: 5.917583593s FAIL (2025-07-18 02:48:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.917623591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............X.X......XX.X..................................] duration: 5.805071958s FAIL (2025-07-18 02:47:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.805097386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XX.......XXXX................................] duration: 6.115331537s FAIL (2025-07-18 02:47:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.115361766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X.X......XX.X.................................] duration: 5.928066142s FAIL (2025-07-18 02:47:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.928100808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............X.X......XX.X................................] duration: 5.790088869s FAIL (2025-07-18 02:46:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.790117544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................XX.......XXXX...............................] duration: 5.864354292s FAIL (2025-07-18 02:46:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.864381543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................X.X......XX.X...............................] duration: 5.757690369s FAIL (2025-07-18 02:45:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.75771737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................XX.......XXXX..............................] duration: 5.854349068s FAIL (2025-07-18 02:45:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854378113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................X.X......XX.X..............................] duration: 5.838188631s FAIL (2025-07-18 02:45:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.838222697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................X.X......XX.X.............................] duration: 5.706906495s FAIL (2025-07-18 02:44:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.706949257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................XX.......XXXX.............................] duration: 5.874769879s FAIL (2025-07-18 02:44:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.874806219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................X.X......XX.X............................] duration: 5.896230502s FAIL (2025-07-18 02:44:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.896256351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XX.......XXXX............................] duration: 6.102194429s FAIL (2025-07-18 02:43:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.102221681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................X.X......XX.X...........................] duration: 5.826297005s FAIL (2025-07-18 02:42:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.826321793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XX.......XXXX...........................] duration: 6.056016447s FAIL (2025-07-18 02:42:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.056040584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................X.X......XX.X..........................] duration: 5.684544408s FAIL (2025-07-18 02:42:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.684573002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XX.......XXXX..........................] duration: 5.875996711s FAIL (2025-07-18 02:41:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.876018542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................X.X......XX.X.........................] duration: 5.835635642s FAIL (2025-07-18 02:41:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.835684636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XX.......XXXX.........................] duration: 5.944006312s FAIL (2025-07-18 02:40:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.944035016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................X.X......XX.X........................] duration: 5.689224932s FAIL (2025-07-18 02:40:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.689254059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................X.X......XX.X.......................] duration: 5.760848521s FAIL (2025-07-18 02:40:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.760878588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XX.......XXXX........................] duration: 5.929862322s FAIL (2025-07-18 02:39:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.929889554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........................X.X......XX.X......................] duration: 5.81603129s FAIL (2025-07-18 02:39:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 8 02:21:35.177845305 (593h11m9.94848664s) [upload=474424]: Uploading segments... Jul 18 02:21:35.177987377 (593h11m9.948628712s) [upload=474424]: Got next segment. Inline: false Jul 18 02:21:35.178033806 (593h11m9.948675141s) [upload=474424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:21:35.178038314 (593h11m9.948679649s) [upload=474424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:21:35.659204645 (593h11m10.42984598s) [seg_pos={0 0}, upload=474424]: Begin upload segment... Jul 18 02:21:35.659248218 (593h11m10.429889543s) [seg_pos={0 0}, upload=474424]: Done begin upload segment. Jul 18 02:21:35.659273166 (593h11m10.429914501s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659300989 (593h11m10.429942324s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659629969 (593h11m10.430271294s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.659876430 (593h11m10.430517765s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660107403 (593h11m10.430748728s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660254844 (593h11m10.43089618s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660432375 (593h11m10.43107371s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660528989 (593h11m10.431170324s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660882566 (593h11m10.431523901s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.660963921 (593h11m10.431605256s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.661213188 (593h11m10.431854513s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.662765930 (593h11m10.433407265s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.663721818 (593h11m10.434363153s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665003170 (593h11m10.435644505s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.665753345 (593h11m10.43639468s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666249704 (593h11m10.436891039s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666615976 (593h11m10.437257311s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:35.666965143 (593h11m10.437606478s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Uploading piece... Jul 18 02:21:36.771152997 (593h11m11.541794332s) [upload=474424]: Next returned nil segment Jul 18 02:21:36.771200578 (593h11m11.541841923s) [upload=474424]: Waiting for error group managing segments... Jul 18 02:21:37.062727209 (593h11m11.833368544s) [noise=true , address=79.127.205.233:4625 , piece_id=WE43MEX62LE3ZHIAOSUJMZZGVOEV2YJN342OJIFRTKLXOI3B3O6A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.234454516 (593h11m12.005095851s) [noise=true , address=79.127.205.235:4673 , piece_id=6K354YXA3H7QJ6W5Y3MWVI353R5YWR6KEJNBBN7UTWIZ7U4AGZ5A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.356875997 (593h11m12.127517372s) [noise=true , address=45.91.95.4:4019 , piece_id=ZA4HDNHYRQSJ55SLOORN2JZCEWAO3TW6BLF4VUTEDTXWVKSGUBDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.415013602 (593h11m12.185654937s) [noise=true , address=45.91.95.2:4022 , piece_id=5DZT7U4BNJRNETPUPGUTMJBFOLDHTZEKFKEPGZI6MPBIQ42MWMDQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.481780768 (593h11m12.252422104s) [noise=true , address=45.91.95.3:4025 , piece_id=OOIXS37H6IYGW7OQFAWVERFDHATX2UASL2QDCIMS5ED2KQPAITKQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.587656953 (593h11m12.358298288s) [noise=true , address=45.91.116.13:4158 , piece_id=VKPPGYBEY2EK7TVTBOCALAZTNJLTX7SXZ23IMA5AZBZYULR7C4RA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.603489976 (593h11m12.374131311s) [noise=true , address=104.36.82.51:4077 , piece_id=P6667ZOVEU66GZA3VHBSZCVRVU3GGEZ6ICWPPGTHXBSZICDIXK2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.632675457 (593h11m12.403316793s) [noise=true , address=45.91.116.6:4001 , piece_id=FAHE4JPUNMAMDXQNDSO26DDXGVGVNQYTAMG7XEAWE3UB5C5JLF7Q, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.633584979 (593h11m12.404226314s) [noise=true , address=79.127.219.38:4021 , piece_id=PVRMXGJHHJQDDSDBTARCNOUOOORY4MZ4524K6LSHI6MNAEBMGFFA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.662224003 (593h11m12.432865338s) [noise=true , address=79.127.219.51:4214 , piece_id=74IMTXLXEVONC6ABLITL3YDEZMOV7TWUZKAWIQBXNXJ7LFJI6TOA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.676693477 (593h11m12.447334802s) [noise=true , address=104.36.82.5:4854 , piece_id=SETPOSULUVRAGK5AB7YUTY6GZJVBAFITMIOI33IPZT6SYJ6UUGFQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.704001817 (593h11m12.474643152s) [noise=true , address=104.36.82.49:4991 , piece_id=WJX5D3GLMRQBLTIYW4CLVKVV22N6W4HR4653KFOHSJ5VPGW33OWQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:21:37.717466809 (593h11m12.488108144s) [noise=true , address=104.36.82.11:4143 , piece_id=FVVMO7VL4UC33C2FL2UA63LCDRUMSOLUEVQTOPVOBXVXTRZ7QRPA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: <nil> Jul 18 02:22:34.473113950 (593h12m9.243755285s) [noise=true , address=77.90.31.12:4602 , piece_id=OS5CDYJA4T3TDZHCHI6U2IWGAAGXYOZS22NXWA526GXKQVLK6E2A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473169977 (593h12m9.243811312s) [noise=true , address=77.90.31.10:4539 , piece_id=WIMAODSQ3QRV7Y7SZN76RAYCARUP32RRQGE2RQNW4KHOPEYSBT4A, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: write tcp 188.166.225.170:59550->77.90.31.10:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473232436 (593h12m9.243873761s) [noise=true , address=77.90.31.21:4887 , piece_id=SGP53S4FGVOF5KTS465L25MJZDYFDGDDAJANNPLU3BSIDPYDO4BA, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473258586 (593h12m9.243899921s) [noise=true , address=77.90.31.18:4840 , piece_id=5WSE2YR4UYZZCP5O4HHF4AFDAEFZTMWNBCC36NH42HOUD2ASJNUQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:22:34.473359119 (593h12m9.244000454s) [noise=true , address=77.90.31.17:4797 , piece_id=OHAWDT4NLGT7URIDDE3NOROCLQCAICQF6RTQNMNJBCFUGETK6DMQ, seg_pos={0 0}, upload=474424]: Done uploading piece. err: ecclient: upload failed (node:12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, address:77.90.31.17:4797): protocol: context deadline exceeded; context deadline exceeded Jul 18 02:22:34.473402903 (593h12m9.244044238s) [seg_pos={0 0}, upload=474424]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:22:34.723019907 (593h12m9.493661232s) [upload=474424]: Done uploading segments... Jul 18 02:22:34.723045165 (593h12m9.4936865s) [upload=474424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.816055787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XX.......XXXX.......................] duration: 5.915713058s FAIL (2025-07-18 02:39:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 3388568s) [noise=true , address=77.90.31.11:4554 , piece_id=6AWUJQ2FYS6GDPCZKMEXY3RI3ZN5P5VZGL2ECHWNWKOMBBGS7NDQ, seg_pos={0 0}, upload=474395]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:51514->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:03.962877852 (593h9m38.733519217s) [seg_pos={0 0}, upload=474395]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 02:20:04.210127302 (593h9m38.980768647s) [upload=474395]: Done uploading segments... Jul 18 02:20:04.210159654 (593h9m38.980800989s) [upload=474395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 18 02:20:24.779878619 (593h9m59.550519954s) [upload=474407]: Starting upload Jul 18 02:20:24.779962249 (593h9m59.550603584s) [upload=474407]: Done starting upload Jul 18 02:20:24.780003679 (593h9m59.550645024s) [upload=474407]: Uploading segments... Jul 18 02:20:24.780157943 (593h9m59.550799278s) [upload=474407]: Got next segment. Inline: false Jul 18 02:20:24.780266601 (593h9m59.550907976s) [upload=474407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:20:24.780275469 (593h9m59.550916824s) [upload=474407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:20:25.031967328 (593h9m59.802608663s) [seg_pos={0 0}, upload=474407]: Begin upload segment... Jul 18 02:20:25.032006182 (593h9m59.802647517s) [seg_pos={0 0}, upload=474407]: Done begin upload segment. Jul 18 02:20:25.032029236 (593h9m59.802670572s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032034727 (593h9m59.802676062s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032066218 (593h9m59.802707543s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032490629 (593h9m59.803131965s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032679040 (593h9m59.803320375s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032750757 (593h9m59.803392092s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.032986588 (593h9m59.803627913s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033008651 (593h9m59.803649976s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033262296 (593h9m59.803903621s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033295740 (593h9m59.803937075s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033520239 (593h9m59.804161574s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033734439 (593h9m59.804375774s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.033953097 (593h9m59.804594432s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034150214 (593h9m59.804791549s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034350338 (593h9m59.804991663s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:25.034544279 (593h9m59.805185604s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Uploading piece... Jul 18 02:20:26.722719428 (593h10m1.493360753s) [upload=474407]: Next returned nil segment Jul 18 02:20:26.722737463 (593h10m1.493378798s) [upload=474407]: Waiting for error group managing segments... Jul 18 02:20:27.069826685 (593h10m1.84046802s) [noise=true , address=45.91.116.5:4298 , piece_id=BX4I6LTLCV36JBDKSG5HRW3KCI5RK5GZSYLNF6UF7JCQV62YSBFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.092893549 (593h10m1.863534884s) [noise=true , address=104.36.82.4:4804 , piece_id=L7QQQYBFIUZ5EAONATMIK4GX4V4SYLTFHJQLKWSXXFCU3JT7E7AQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.143535552 (593h10m1.914176887s) [noise=true , address=45.91.116.6:4973 , piece_id=GNALGJMD5QZS4UAO3VKUIIMNRUT5E5CFYFCNHFCMW4I53C43GQ5Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.151311830 (593h10m1.921953165s) [noise=true , address=45.91.116.4:4938 , piece_id=L7KPVBMUZNY5RZ67SGQRZAE4OPQCTD65A2HWCN6EHQ6NW5TRACFA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.154544595 (593h10m1.92518593s) [noise=true , address=45.91.116.8:4048 , piece_id=LWZWKGS2IIPE4TOXZIHWF5LTW3G5R6LL4EYNOBLG3LUGX76PZRDQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.167648668 (593h10m1.938290003s) [noise=true , address=104.36.82.8:4970 , piece_id=QYLXIOE3XZQIWLLCKLYCG6Q5ZYYSBOECYWENAF5T5L5UU7EB7WWA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210335119 (593h10m1.980976454s) [noise=true , address=104.36.82.2:4731 , piece_id=NVBPEEVKXWUXXJ27Y4IFX4IO4ICGN4LXNELKUIE3ZVJX4VL2LD4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.210500045 (593h10m1.98114138s) [noise=true , address=45.91.116.5:4297 , piece_id=I3FBBP7NI4RH5VTKFUWB5L3EH7BERV2CT2DEENVTV3X4KCOFLM4Q, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.232506489 (593h10m2.003147824s) [noise=true , address=45.91.116.14:4189 , piece_id=HOFJMFCK3S2T3YBUPIPTCY227U4E7XQASACJOF464MP6NFJK56DQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.234721757 (593h10m2.005363092s) [noise=true , address=104.36.82.13:4186 , piece_id=S5H5OLN7B5YA6S2JKVDMCP5UI2C5RCNXUBGBFIHPGGYMFIS522SA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.238800071 (593h10m2.009441406s) [noise=true , address=104.36.82.6:4873 , piece_id=SH6NJFVS2IJ5QMLO6WJURRMSWJAXD46K7UFPGYUEZQ25LYTSODPA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:27.254974146 (593h10m2.025615481s) [noise=true , address=104.36.82.16:4293 , piece_id=PS52OGX2Q4G2AIMOFVN2UAKWC6XTWDJ5TGK6CTYALBSJLMRT3BRA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: <nil> Jul 18 02:20:54.085289022 (593h10m28.855930357s) [noise=true , address=77.90.31.18:4806 , piece_id=E5IYEE2FVEMMHHV7YVDWUFQDZMDRDKATHTMFBLIYBR344KI3FTYQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:53138->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085367181 (593h10m28.856008517s) [noise=true , address=77.90.31.10:4519 , piece_id=WEZBWBRN3GPIC6QPXNTM5LPSV6UKUXLDGWWG7HOXXBXOY7KEWYOQ, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:35864->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085427958 (593h10m28.856069293s) [noise=true , address=77.90.31.14:4661 , piece_id=TXO55SI4F2VW356XSS4M3QSN57WW7I2YV65BW3GCLAWKTUBAN44A, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085460200 (593h10m28.856101535s) [noise=true , address=77.90.31.17:4771 , piece_id=UXQMWPDAZP7UEHH4AEMOXBNLVZHMMX5VPMDVWHQWK24NXMVY7MYA, seg_pos={0 0}, upload=474407]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:20:54.085487322 (593h10m28.856128657s) [seg_pos={0 0}, upload=474407]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 02:20:54.335312937 (593h10m29.105954273s) [upload=474407]: Done uploading segments... Jul 18 02:20:54.335338926 (593h10m29.105980261s) [upload=474407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.915737254s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)