Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.794999575s FAIL (2025-07-19 20:30:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.795026528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.84367248s FAIL (2025-07-19 20:29:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.843704742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 5.814676399s FAIL (2025-07-19 20:29:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.814714642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 5.732395483s FAIL (2025-07-19 20:28:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.732437773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 5.987719467s FAIL (2025-07-19 20:28:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.987748311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 5.844902546s FAIL (2025-07-19 20:28:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.844941851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 5.948126196s FAIL (2025-07-19 20:27:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.948144732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 5.789732241s FAIL (2025-07-19 20:27:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.789754883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 5.727603364s FAIL (2025-07-19 20:27:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.727629885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 5.95236794s FAIL (2025-07-19 20:27:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.952394992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 5.795260433s FAIL (2025-07-19 20:26:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.795287877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration: 5.691636098s FAIL (2025-07-19 20:26:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.69166879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 5.923729683s FAIL (2025-07-19 20:25:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.923760051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 5.84951583s FAIL (2025-07-19 20:25:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.849550756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 5.803356923s FAIL (2025-07-19 20:25:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.803383214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXXXXX.....................................................] duration: 6.208515159s FAIL (2025-07-19 20:25:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.208543365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 5.859913827s FAIL (2025-07-19 20:24:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.8599414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 5.877566691s FAIL (2025-07-19 20:24:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.877591459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXXXXX....................................................] duration: 5.947244379s FAIL (2025-07-19 20:24:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.947272173s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXXXXX...................................................] duration: 5.869367283s FAIL (2025-07-19 20:23:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.869396409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XXXXXX..................................................] duration: 6.008727384s FAIL (2025-07-19 20:23:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.008752681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 6.041555832s FAIL (2025-07-19 20:23:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.041584077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XXXXXX.................................................] duration: 6.033077873s FAIL (2025-07-19 20:22:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.033106688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 5.643638312s FAIL (2025-07-19 20:21:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.643667459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XXXXXX................................................] duration: 5.948020451s FAIL (2025-07-19 20:21:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.948052192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XXXXXX...............................................] duration: 6.329065677s FAIL (2025-07-19 20:21:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.329094993s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 5.724865504s FAIL (2025-07-19 20:20:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.724892365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XXXXXX..............................................] duration: 5.989981111s FAIL (2025-07-19 20:20:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.990031227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XXXXXX.............................................] duration: 5.87360847s FAIL (2025-07-19 20:20:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.873639679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 5.776204982s FAIL (2025-07-19 20:19:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.77623544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XXXXXX............................................] duration: 5.929649605s FAIL (2025-07-19 20:19:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.929688268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 5.888057626s FAIL (2025-07-19 20:19:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.888082705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XXXXXX...........................................] duration: 5.851410881s FAIL (2025-07-19 20:18:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.851442892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 5.604662532s FAIL (2025-07-19 20:18:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.604684163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XXXXXX..........................................] duration: 5.93996125s FAIL (2025-07-19 20:18:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.939987399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XXXXXX.........................................] duration: 5.786898695s FAIL (2025-07-19 20:17:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.786922319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration: 5.544238609s FAIL (2025-07-19 20:17:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.544257587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XXXXXX........................................] duration: 5.965177909s FAIL (2025-07-19 20:17:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.965205712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 5.680441668s FAIL (2025-07-19 20:17:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.680471545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XXXXXX.......................................] duration: 5.877487823s FAIL (2025-07-19 20:16:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.877513734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 5.621008196s FAIL (2025-07-19 20:16:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.621042231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 5.80495373s FAIL (2025-07-19 20:16:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.804976173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................XXXXXX......................................] duration: 6.049248582s FAIL (2025-07-19 20:15:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.049271397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................XXXXXX.....................................] duration: 6.04149987s FAIL (2025-07-19 20:15:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.041527695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 5.591861537s FAIL (2025-07-19 20:15:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.591891063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 5.666965449s FAIL (2025-07-19 20:14:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.666988113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................XXXXXX....................................] duration: 5.910128805s FAIL (2025-07-19 20:14:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.910201504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 5.728246978s FAIL (2025-07-19 20:13:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.728269632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XXXXXX...................................] duration: 5.945745589s FAIL (2025-07-19 20:13:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== (needed at least 13 but got 8) Jul 19 19:45:56.955947272 (20h55m3.328526617s) [upload=16640]: Starting upload Jul 19 19:45:56.956012336 (20h55m3.328591681s) [upload=16640]: Done starting upload Jul 19 19:45:56.956155420 (20h55m3.328734775s) [upload=16640]: Uploading segments... Jul 19 19:45:56.956460032 (20h55m3.329039378s) [upload=16640]: Got next segment. Inline: false Jul 19 19:45:56.956489009 (20h55m3.329068354s) [upload=16640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 19:45:56.956494329 (20h55m3.329073684s) [upload=16640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 19:45:57.188815182 (20h55m3.561394537s) [seg_pos={0 0}, upload=16640]: Begin upload segment... Jul 19 19:45:57.188859768 (20h55m3.561439123s) [seg_pos={0 0}, upload=16640]: Done begin upload segment. Jul 19 19:45:57.188881990 (20h55m3.561461345s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.188916005 (20h55m3.56149536s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189230216 (20h55m3.561809571s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189362509 (20h55m3.561941864s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189500263 (20h55m3.562079618s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189661682 (20h55m3.562241037s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.189944903 (20h55m3.562524258s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190136841 (20h55m3.562716196s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.190521236 (20h55m3.563100591s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191050038 (20h55m3.563629413s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191561276 (20h55m3.564140641s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.191980658 (20h55m3.564560023s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192371286 (20h55m3.564950651s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.192693993 (20h55m3.565273348s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193026399 (20h55m3.565605754s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:57.193341352 (20h55m3.565920707s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Uploading piece... Jul 19 19:45:58.904082428 (20h55m5.276661783s) [upload=16640]: Next returned nil segment Jul 19 19:45:58.904100943 (20h55m5.276680298s) [upload=16640]: Waiting for error group managing segments... Jul 19 19:45:59.240386913 (20h55m5.612966268s) [noise=true , address=104.36.82.5:4837 , piece_id=KWDKXHG7BRJBGJJA4QECLAXFNWUTBB56U4XCJXLXEI4HMNTBP7SA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.265222572 (20h55m5.637801917s) [noise=true , address=104.36.82.16:4294 , piece_id=5BUSMFOTB2PEWUEZ6A54VBTLGFYLEMCMJ3JS2K2UJDW6BAIYMNKA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.328187464 (20h55m5.700766819s) [noise=true , address=104.36.82.14:4223 , piece_id=FHL3KR2FWK3UN3RGKJR2ASC2MOLXXFZD7SUATGKN7KTRDQBKANCQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.340208313 (20h55m5.712787668s) [noise=true , address=104.36.82.15:4259 , piece_id=ZV52K26NYDFEG44J57VLLATELCMLPLXH7FQ7OBGPQMW2QGGQ42SQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.388879266 (20h55m5.761458621s) [noise=true , address=104.36.82.10:4077 , piece_id=Y2HUXNCWXIVK2KHJNXM7MSJV3I4MFGXWRWEK6GHJ3XJPCHHMO7JQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.425886182 (20h55m5.798465537s) [noise=true , address=104.36.82.8:4972 , piece_id=H4JTIHR4YWG6JWOUWLZB3XSDRQOHHUMPSVIN6BNGTZ3NQWEIDIYQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.437905088 (20h55m5.810484443s) [noise=true , address=104.36.82.7:4910 , piece_id=JFHPTGV4Z6NUH6HXFOXOT24GYVHZKNQETH2SKQWVFAG7KBOPRJ5Q, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:45:59.453460110 (20h55m5.826039455s) [noise=true , address=104.36.82.11:4115 , piece_id=TNDZSLUAJFAWM6CAVRDK35LAXDEYYNLQNEA3NKUIGQ77REKYKHTA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: <nil> Jul 19 19:46:26.316383271 (20h55m32.688962626s) [noise=true , address=45.91.116.13:4153 , piece_id=75OGJ56DQMKOKPIW4G7H5DDVXCSKNMPNFH4O6N7C2OYLR4WGIJLA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316453304 (20h55m32.68903266s) [noise=true , address=77.90.31.12:4590 , piece_id=UUPP7J6SD5CIWXNZJ3RSZ24OADTUIXNTBCEI5WXH3MXB2DVH5IJQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:34108->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316503360 (20h55m32.689082716s) [noise=true , address=45.91.116.4:4938 , piece_id=34E2KP6C2YVJJZ5ZS2NQCGZ4EXXERK75YNDUYEURNS5ZQ5ZPQDNA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 188.166.225.170:50430->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316554318 (20h55m32.689133683s) [noise=true , address=45.91.116.7:4010 , piece_id=ADVPKMIIYWWZ5MLTJYDZRBELVOHSQT3USU3XVQXMVUTL46NAZKTQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: rpc: dial tcp 45.91.116.7:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316571071 (20h55m32.689150416s) [noise=true , address=77.90.31.11:4555 , piece_id=CZMSPHUHSKZBM4ZFAIXMLEFFR7GGEUXS47TDNSJ6URLKBRJHIMQA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316611477 (20h55m32.689190832s) [noise=true , address=77.90.31.15:4698 , piece_id=P4W553EH243TPBG2VTMCUI24VJCRBCMRBIWB3AX6OC42C6Q6QTOQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316617990 (20h55m32.689197335s) [noise=true , address=45.91.116.7:4009 , piece_id=IO63DS5IXXTUELN2NI6KIA2NIBOC6WHPUIQFJBKWM7DYITX6ELHQ, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316646504 (20h55m32.689225849s) [noise=true , address=45.91.116.6:4973 , piece_id=4ZUD4MCUNWBCL6EAXSITRFCS5FMK6ZTDGLEFSEFDWXOKRPAX6NHA, seg_pos={0 0}, upload=16640]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:26.316660982 (20h55m32.689240337s) [seg_pos={0 0}, upload=16640]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 19 19:46:26.546689890 (20h55m32.919269245s) [upload=16640]: Done uploading segments... Jul 19 19:46:26.546711271 (20h55m32.919290626s) [upload=16640]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.945763625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 5.71591356s FAIL (2025-07-19 20:13:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7848 (20h54m48.288967213s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916415812 (20h54m48.288995167s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916709062 (20h54m48.289288417s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916861384 (20h54m48.289440749s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.916935646 (20h54m48.289515001s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917160236 (20h54m48.289739591s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917175124 (20h54m48.289754479s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917373244 (20h54m48.289952599s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917481661 (20h54m48.290061016s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917615176 (20h54m48.290194531s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.917784180 (20h54m48.290363535s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918031533 (20h54m48.290610888s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918281632 (20h54m48.290860987s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918492254 (20h54m48.291071609s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918719600 (20h54m48.291298955s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.918930743 (20h54m48.291510098s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:41.919150143 (20h54m48.291729498s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Uploading piece... Jul 19 19:45:43.033526636 (20h54m49.406105991s) [upload=16638]: Next returned nil segment Jul 19 19:45:43.033548608 (20h54m49.406127963s) [upload=16638]: Waiting for error group managing segments... Jul 19 19:45:43.368051177 (20h54m49.740630572s) [noise=true , address=79.127.205.233:4621 , piece_id=35MW6WACUECIHLE3W5WWCGHX5IZ2YYLISRHMTORWNK4YM7JUUM5A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.667078612 (20h54m50.039657967s) [noise=true , address=79.127.205.245:4245 , piece_id=ABTIC4DMOPNTKUJ7YYO2T6XRKFU6YOP77SIQEWIPRI4IRDPJRWGA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.830458308 (20h54m50.203037673s) [noise=true , address=79.127.201.215:4358 , piece_id=FWGZX5AID6PFEKW75LU2CRSJ7NTF5NSKNFA2UNV57V6ZYFDRCJNQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.839119359 (20h54m50.211698754s) [noise=true , address=104.36.82.18:4400 , piece_id=FQNSBCA37YGIBG4NIUUJE4CVIFQIFUAFXG2SXI57VB6ZE5ZLSVEQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.846613288 (20h54m50.219192663s) [noise=true , address=104.36.82.40:4961 , piece_id=VPUE2QBXZUSEZ27STEH7HMUN4V5O2RZRS572QD2KLRGVO7PDATEA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.905336063 (20h54m50.277915438s) [noise=true , address=79.127.219.47:4055 , piece_id=5SCFP4WKEX327LZ6OTGOBP4L3VCDECHT7YVKZ7PRQB2LHDEVMUDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.946328965 (20h54m50.31890832s) [noise=true , address=104.36.82.44:4902 , piece_id=I53357NLQDIKXESB6MRNKLQCB2QLXQE7JXGRIKZUOQWALQXHENCA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953031279 (20h54m50.325610624s) [noise=true , address=104.36.82.12:4152 , piece_id=EQFL55U6R6ELK6AUIJTCFKPQVVVSLG3DXLZTOM56KHYXUQLXV6PA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:43.953713835 (20h54m50.32629319s) [noise=true , address=104.36.82.49:4980 , piece_id=ZHVON73WV3JXGECZ5IMAEHTKFRDHZYUROS7SL4FOLVO2IZ2YH4AQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:45:44.164465048 (20h54m50.537044403s) [noise=true , address=104.36.82.51:4071 , piece_id=QJ7TRQ7GJCTPUVJEOXKZ7KQ7IK4GKZR3MULUBYUU7FLC6JSJXDWA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: <nil> Jul 19 19:46:40.808104090 (20h55m47.180683445s) [noise=true , address=45.91.116.9:4290 , piece_id=MNL77IA22542NU543FL6UAV2XWBSR6N3TGWK2DQEN7ZZAU56J66A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: rpc: dial tcp 45.91.116.9:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808164726 (20h55m47.180744081s) [noise=true , address=45.91.116.14:4212 , piece_id=MQ67AOPO6WCZ6TPN3LWPUJ6ODJKVZWOMD6ACOL345H7IRLKDAFRA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, address:45.91.116.14:4212): protocol: write tcp 188.166.225.170:60452->45.91.116.14:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808201998 (20h55m47.180781343s) [noise=true , address=77.90.31.18:4834 , piece_id=BCPFOSJZCAMP4RNNCL64TYC5AMCWXCB5GMP6FW5EF66Y3JT6INDA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: write tcp 188.166.225.170:39120->77.90.31.18:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808320064 (20h55m47.180899419s) [noise=true , address=77.90.31.16:4740 , piece_id=FFROTB5MBFFGCM2ETZ2QK3IR6Y2Y2LB3VNZPCNQ5NCHVOWYXU56A, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 188.166.225.170:48266->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808338409 (20h55m47.180917764s) [noise=true , address=77.90.31.17:4774 , piece_id=PZSNV4UQDV6OLR4PV3APVEJMK3QFSAA2TFTI3Q5TVX5UX5AJXKHQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:57650->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808386862 (20h55m47.180966217s) [noise=true , address=77.90.31.21:4900 , piece_id=SPFBTE2TWSS6EZALXQ4HCWX67PPYNEN3IN3VZV52NVZONEOYV6EA, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 188.166.225.170:55570->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808398804 (20h55m47.18097816s) [noise=true , address=77.90.31.10:4530 , piece_id=YHCHPBJU4TNX4BN3UB2M5ALIWMYCBASWDEXXE2I7SVNG4ZAN5ETQ, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: rpc: dial tcp 77.90.31.10:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808427980 (20h55m47.181007335s) [noise=true , address=77.90.31.15:4701 , piece_id=ZBOEDVZ5TJI2SXFIRFYBAN55VUD6WFBPP33VFZD4JI4ODTBKPD3Q, seg_pos={0 0}, upload=16638]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 188.166.225.170:55032->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 19:46:40.808454491 (20h55m47.181033836s) [seg_pos={0 0}, upload=16638]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 19 19:46:41.038976382 (20h55m47.411555737s) [upload=16638]: Done uploading segments... Jul 19 19:46:41.039000268 (20h55m47.411579623s) [upload=16638]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.715937125s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)