Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.505111656s FAIL (2025-10-23 12:04:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.505142586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 7.442591717s FAIL (2025-10-23 12:04:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.442618197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 12.199973669s FAIL (2025-10-23 12:04:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.200004269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.029592826s FAIL (2025-10-23 12:03:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.029628363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.317597417s FAIL (2025-10-23 12:03:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.317645038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.954963807s FAIL (2025-10-23 12:03:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.955008573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.157213476s FAIL (2025-10-23 12:02:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.157246077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.90360056s FAIL (2025-10-23 12:02:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 7.903628804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 7.356288758s FAIL (2025-10-23 12:02:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.356314398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.056772223s FAIL (2025-10-23 12:02:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.056804314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.248440875s FAIL (2025-10-23 12:01:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.248475521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.021364636s FAIL (2025-10-23 12:01:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.021411307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.07679113s FAIL (2025-10-23 12:01:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.076824473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.986832941s FAIL (2025-10-23 12:00:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.986866484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.032088219s FAIL (2025-10-23 12:00:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.032122232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.077361748s FAIL (2025-10-23 12:00:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.077464163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.99583826s FAIL (2025-10-23 11:59:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.995866244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.908566222s FAIL (2025-10-23 11:59:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.908604305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.961934183s FAIL (2025-10-23 11:59:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.961963649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.191377167s FAIL (2025-10-23 11:58:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.191408235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.920834516s FAIL (2025-10-23 11:58:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.920861919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.143325084s FAIL (2025-10-23 11:58:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.143361354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.907053324s FAIL (2025-10-23 11:58:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.907084113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.024242549s FAIL (2025-10-23 11:57:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.024276173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.973179558s FAIL (2025-10-23 11:57:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.973224224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.92759641s FAIL (2025-10-23 11:57:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.92762771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.919221921s FAIL (2025-10-23 11:56:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.919257458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.969701839s FAIL (2025-10-23 11:56:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.969754148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.168630863s FAIL (2025-10-23 11:55:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.168661271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.028420473s FAIL (2025-10-23 11:55:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.028452644s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.22802324s FAIL (2025-10-23 11:54:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.228053658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.197101961s FAIL (2025-10-23 11:54:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.197130115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.919329367s FAIL (2025-10-23 11:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.919368401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.987562683s FAIL (2025-10-23 11:54:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.987601226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.065866717s FAIL (2025-10-23 11:53:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.065895443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.020934139s FAIL (2025-10-23 11:52:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.020969927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.895514151s FAIL (2025-10-23 11:52:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.895542264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.23777058s FAIL (2025-10-23 11:52:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.237798574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.966255856s FAIL (2025-10-23 11:52:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.966289689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.00263027s FAIL (2025-10-23 11:51:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.002662972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.012622298s FAIL (2025-10-23 11:51:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.012658948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.719800404s FAIL (2025-10-23 11:50:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.719834377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.00817073s FAIL (2025-10-23 11:50:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.0082072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.892439555s FAIL (2025-10-23 11:50:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.89290422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.379356534s FAIL (2025-10-23 11:49:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.379388435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.014175093s FAIL (2025-10-23 11:48:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.014204088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 8.024571933s FAIL (2025-10-23 11:48:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.024603624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.255643029s FAIL (2025-10-23 11:48:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.255688417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.899240647s FAIL (2025-10-23 11:47:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690456804 (718h0m34.152549287s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.690872646 (718h0m34.152965129s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691285382 (718h0m34.153377855s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.691668792 (718h0m34.153761275s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692051451 (718h0m34.154143934s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.692519563 (718h0m34.154612046s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693109987 (718h0m34.15520247s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.693627412 (718h0m34.155719895s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695208062 (718h0m34.157300535s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.695245022 (718h0m34.157337495s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.696963003 (718h0m34.159055486s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.699114781 (718h0m34.161207274s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:48.701035708 (718h0m34.163128191s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Uploading piece... Oct 23 11:31:50.481046142 (718h0m35.943138625s) [upload=631616]: Next returned nil segment Oct 23 11:31:50.481069187 (718h0m35.94316168s) [upload=631616]: Waiting for error group managing segments... Oct 23 11:31:50.827047413 (718h0m36.289139896s) [noise=true , address=45.91.116.13:4154 , piece_id=K4LAX4ARUTWZVUIFJYWYKNYQY3JGJ2SKU6FNZBYCCAYXSYGG4JVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.871438262 (718h0m36.333530755s) [noise=true , address=77.90.31.15:4699 , piece_id=4FYFKFWOAHLCOOS2JKCOV4PRCUAZZ7RMHQ7ZZJGOHA64UT2Y43BA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:31:50.927316536 (718h0m36.389409009s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTZE5IDTGZQWT76K2EP6UBILYPU3CZ6SUNSYAMAZPJ26A4I6HYA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: <nil> Oct 23 11:32:17.743342216 (718h1m3.205434689s) [noise=true , address=145.79.227.16:4909 , piece_id=ECDDS6LEWRO5PP5LBHTX7DVYWH2VJIVNLZ2VIJLTJR4E4R6OZ27A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743421767 (718h1m3.20551426s) [noise=true , address=145.79.227.3:4149 , piece_id=S7YYQ6LAMGGMHDD72FGKSZPHB7RQ32OQGBAK2USILZTA4K7AYTRA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743578346 (718h1m3.205670829s) [noise=true , address=145.79.227.18:4005 , piece_id=EXGEPKKFW437NQKWSVDGLYLXKYDLSXJUX73IADWRMUN4EXFVENVA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743666102 (718h1m3.205758595s) [noise=true , address=145.79.227.8:4329 , piece_id=BGYJYJ3DQHNTBEO5VGB7DDX7O5ORMMV2QEQWGW6N2YVDAH2U7ANQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: rpc: dial tcp 145.79.227.8:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743773477 (718h1m3.20586596s) [noise=true , address=145.79.227.19:4078 , piece_id=WK4WKEYW3M5JN2WABQDKI4AWI7HVOLCB57OXZINVC3D7VT6XOVHQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743818393 (718h1m3.205910876s) [noise=true , address=145.79.227.12:4765 , piece_id=FIAFAU6I54WOBESSXR74YPBVVGB4JCAOV3V3JGOXUF5TI2LP5VYQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743949432 (718h1m3.206041915s) [noise=true , address=145.79.227.9:4366 , piece_id=ITJARDLREX2M3O2Y56FHXL7CN4VMJVGLC52TWZFPFU27LBJ6RIXA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.743967877 (718h1m3.20606036s) [noise=true , address=145.79.227.22:4509 , piece_id=FZZLTKVAGZHVVAFNEX3VVARZINI6BBKLSULSL5UV6SFWGQENCLSQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:54808->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744058890 (718h1m3.206151373s) [noise=true , address=145.79.227.15:4873 , piece_id=G236QF5XILOYRRS6XZGEZLZZRCYPQA5PF5OAMMDPH7EG52LKGEOA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: read tcp 188.166.225.170:57766->145.79.227.15:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744152779 (718h1m3.206245262s) [noise=true , address=145.79.227.5:4221 , piece_id=3MLV7IAQBHC4TNTO6Z6MU63EJULUAIIIFRSZIKEDNOEIBYQWRG2A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: read tcp 188.166.225.170:58548->145.79.227.5:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744170483 (718h1m3.206262966s) [noise=true , address=145.79.227.10:4402 , piece_id=BVKFMYUIYQGUBSBDLSAYWHZ7HTRPYRRWXLFDA63WDIUBDKPX3C3A, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:53316->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744261245 (718h1m3.206353719s) [noise=true , address=145.79.227.13:4803 , piece_id=OFIK5J2DJ4FYSDXETP5LN2HN5D2DVZSZODPHEHD5YRBC4OCOS6PA, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:60182->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744294208 (718h1m3.206386691s) [noise=true , address=145.79.227.14:4839 , piece_id=VZSETJNSU54VHHXSGF6EJ3QP76WYTNEKY4GSXQSKIW6OFHUNYNKQ, seg_pos={0 0}, upload=631616]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:34924->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:17.744321941 (718h1m3.206414424s) [seg_pos={0 0}, upload=631616]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Oct 23 11:32:17.985568445 (718h1m3.447660938s) [upload=631616]: Done uploading segments... Oct 23 11:32:17.985602079 (718h1m3.447694562s) [upload=631616]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.899277999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.976698919s FAIL (2025-10-23 11:47:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Oct 23 11:31:31.025502072 (718h0m16.487594545s) [upload=631613]: Got next segment. Inline: false Oct 23 11:31:31.025532009 (718h0m16.487624492s) [upload=631613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 11:31:31.025536568 (718h0m16.487629051s) [upload=631613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 11:31:31.293502040 (718h0m16.755594524s) [seg_pos={0 0}, upload=631613]: Begin upload segment... Oct 23 11:31:31.293576993 (718h0m16.755669476s) [seg_pos={0 0}, upload=631613]: Done begin upload segment. Oct 23 11:31:31.293614254 (718h0m16.755706737s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.293645163 (718h0m16.755737646s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294377137 (718h0m16.75646962s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.294744866 (718h0m16.756837339s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295060107 (718h0m16.75715259s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295523449 (718h0m16.757615933s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.295972976 (718h0m16.758065459s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.296650265 (718h0m16.758742748s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297332184 (718h0m16.759424667s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.297776991 (718h0m16.759869464s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.298758580 (718h0m16.760851063s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.299787059 (718h0m16.761879542s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.301378729 (718h0m16.763471212s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.303181892 (718h0m16.765274386s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.304832666 (718h0m16.76692516s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.308342470 (718h0m16.770434953s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.309207536 (718h0m16.771300039s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:31.313177056 (718h0m16.775269549s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Uploading piece... Oct 23 11:31:32.878794349 (718h0m18.340886822s) [upload=631613]: Next returned nil segment Oct 23 11:31:32.878862920 (718h0m18.340955403s) [upload=631613]: Waiting for error group managing segments... Oct 23 11:31:33.283194573 (718h0m18.745287076s) [noise=true , address=77.90.31.11:4588 , piece_id=E33W5T24WAKDF4L3NXCXKGWQLZOOEKQNXFUGQLUIWHE7VQ3HR6CQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.285852094 (718h0m18.747944577s) [noise=true , address=77.90.31.18:4815 , piece_id=E7MWA3IGWCW2HQF4GUEQAEFI2YX5LFAH4245PB3HO2X24PMDZZWQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.316844890 (718h0m18.778937373s) [noise=true , address=45.91.116.7:4040 , piece_id=U3V4FFZGFWCMOGFT6ACAFTKZODOB5YEUYHKX2OWEFJSXXXP7D7UQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.356409389 (718h0m18.818501872s) [noise=true , address=45.91.116.12:4239 , piece_id=6B4XU3WRYHQ3GX2QV5K4Q6EWG6LDGCWPBSKXY3BC7TA643CX55EA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.358993130 (718h0m18.821085613s) [noise=true , address=77.90.31.19:4868 , piece_id=TVWXIIYN7SLHY4EMSG3ZXRMJLQ4EY536FPNP6T26CNRJ45OS6Z5Q, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.384622624 (718h0m18.846715107s) [noise=true , address=77.90.31.10:4529 , piece_id=HNWS6IDOXLH2J2LIVEEJWUIKY4HVHHFJ5KNQP4IA2WLMN5WMX4TA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.393367376 (718h0m18.855459859s) [noise=true , address=77.90.31.16:4753 , piece_id=TTJNKLCSAN32AV3MOAVVFQCZ3M24WTWWD4TCGHF2QJK5ZJ2AL6UA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.416895519 (718h0m18.878988002s) [noise=true , address=45.91.116.5:4315 , piece_id=2PZSVYUTILRUZVG5ZRNJQPJQITKS4PJUXJLETIII66YRCP7KBQLQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.425298731 (718h0m18.887391214s) [noise=true , address=77.90.31.17:4799 , piece_id=OGDGC7DAPTFCSXG53MTLPBIWLTFSYAMFICAELV7DXPZWQ7JNMOQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.458527425 (718h0m18.920619898s) [noise=true , address=45.91.116.11:4143 , piece_id=LECTGB4DNSNTIPQ7HF7GLOG4JMW7GULJNBX5MKTEKCE6D32XHPQA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.496510443 (718h0m18.958602916s) [noise=true , address=45.91.116.9:4283 , piece_id=Y3F7FJKAW4UIPKMXIAFIOIYDX4BS3X3LGLDHODL72D6T6B7DDO4A, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.529020308 (718h0m18.991112781s) [noise=true , address=77.90.31.15:4729 , piece_id=I7LYRJ67NFLNYE6PMDABV2M7CSK2UJ3MD5NYCQGH73JDP27TSUUA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:31:33.593192024 (718h0m19.055284507s) [noise=true , address=77.90.31.21:4889 , piece_id=UWNFS4JJ4OCTOZPISKOJWTPTDII4AGV6GXOLDGYUEFZXY46HZUPA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: <nil> Oct 23 11:32:30.330619309 (718h1m15.792711792s) [noise=true , address=145.79.227.50:4401 , piece_id=EFKR4NN6SJDXQFZNLGCTZGIHDKU6P7JLZ63SHTJTWLLOVKXCPAAQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12p25affFUZgCByWE2j1FS5k7tedvZsHeCcmwMeerMXPFNf8eEY, address:145.79.227.50:4401): protocol: read tcp 188.166.225.170:32946->145.79.227.50:4401: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330696065 (718h1m15.792788538s) [noise=true , address=145.79.227.47:4285 , piece_id=UVJFA7REUZLWGZYEYHXB4CFK6UGXEL5VVGWRZOBI44Q5GTENWGHA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:145.79.227.47:4285): protocol: read tcp 188.166.225.170:54628->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330771259 (718h1m15.792863742s) [noise=true , address=145.79.227.40:4062 , piece_id=4ACVXAWBV72FO7XKHNUCBKCZ5GGGYZ3H3GO4LIEFEKABJIMWMRBA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 23 11:32:30.330817136 (718h1m15.792909609s) [noise=true , address=145.79.227.29:4947 , piece_id=UUHAP7U5ZYCVRDTYQRS7RANJ7HBWWRTKDD4LFNQPPAXZH6EJGISA, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:145.79.227.29:4947): protocol: context deadline exceeded; context deadline exceeded Oct 23 11:32:30.330934990 (718h1m15.793027473s) [noise=true , address=145.79.227.31:4822 , piece_id=Z7A5GY7I5JTECZSSNKGM6K6LKZH7ABTS5ONYWO4ZVF6TGMZNMROQ, seg_pos={0 0}, upload=631613]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:145.79.227.31:4822): protocol: read tcp 188.166.225.170:46850->145.79.227.31:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Oct 23 11:32:30.330967091 (718h1m15.793059564s) [seg_pos={0 0}, upload=631613]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 23 11:32:30.575020441 (718h1m16.037112924s) [upload=631613]: Done uploading segments... Oct 23 11:32:30.575050358 (718h1m16.037142851s) [upload=631613]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.976726963s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)